Sep 08 18:34:24.196646 ip-10-0-132-43 systemd[1]: Starting Kubernetes Kubelet... Sep 08 18:34:24.531290 ip-10-0-132-43 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 18:34:24.531290 ip-10-0-132-43 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 08 18:34:24.531290 ip-10-0-132-43 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 18:34:24.531290 ip-10-0-132-43 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 08 18:34:24.531290 ip-10-0-132-43 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 18:34:24.532904 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.532823 2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 08 18:34:24.536458 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536443 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 18:34:24.536458 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536457 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 18:34:24.536458 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536461 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536464 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536467 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536481 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536484 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536488 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536490 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536493 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536496 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536498 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536501 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536504 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536506 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536509 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536511 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536517 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536521 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536524 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536527 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 18:34:24.536549 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536529 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536533 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536535 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536538 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536541 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536543 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536546 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536549 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536552 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536555 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536559 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536562 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536565 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536567 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536570 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536574 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536577 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536580 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536582 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536585 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 18:34:24.536999 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536588 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536590 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536593 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536596 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536599 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536601 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536604 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536607 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536609 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536612 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536614 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536616 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536619 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536622 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536625 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536627 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536630 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536632 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536635 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536638 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 18:34:24.537498 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536640 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536643 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536645 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536648 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536651 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536654 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536656 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536659 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536661 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536664 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536667 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536669 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536671 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536674 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536677 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536679 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536682 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536684 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536687 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536690 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 18:34:24.538013 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536692 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536695 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536697 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536700 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536702 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536706 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536709 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536711 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536714 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536717 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536719 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536724 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536728 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536731 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536735 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536739 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536742 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.536745 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537485 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 18:34:24.538529 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537494 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537497 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537500 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537503 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537506 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537508 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537511 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537514 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537517 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537520 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537522 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537525 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537527 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537530 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537533 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537535 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537538 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537540 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537543 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537546 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 18:34:24.539008 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537549 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537552 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537554 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537557 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537567 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537571 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537574 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537577 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537580 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537585 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537589 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537591 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537594 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537596 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537599 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537601 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537604 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537607 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537610 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537612 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 18:34:24.539526 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537615 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537617 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537620 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537622 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537625 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537627 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537630 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537632 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537635 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537637 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537639 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537643 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537647 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537651 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537654 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537656 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537659 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537661 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537664 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537666 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 18:34:24.540014 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537669 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537672 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537674 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537680 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537683 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537685 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537688 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537690 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537693 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537696 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537699 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537701 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537703 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537706 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537709 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537711 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537714 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537716 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537719 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 18:34:24.540521 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537721 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537724 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537728 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537731 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537734 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537737 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537740 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537743 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537746 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537748 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537751 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537754 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537757 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537759 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537762 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537764 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537767 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537772 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.537776 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 18:34:24.540980 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538173 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538178 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538181 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538184 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538187 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538189 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538192 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538195 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538197 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538200 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538202 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538205 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538208 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538210 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538213 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538216 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538218 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538221 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538224 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538227 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 18:34:24.541455 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538230 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538232 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538234 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538237 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538240 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538242 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538244 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538247 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538249 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538252 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538254 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538257 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538260 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538262 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538264 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538267 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538269 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538272 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538275 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538277 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 18:34:24.541939 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538279 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538282 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538285 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538287 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538290 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538294 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538298 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538301 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538304 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538307 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538309 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538312 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538315 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538317 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538320 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538322 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538325 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538327 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538329 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 18:34:24.542480 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538332 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538335 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538337 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538340 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538342 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538345 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538348 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538351 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538354 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538357 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538373 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538376 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538378 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538381 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538383 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538386 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538389 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538391 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538394 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538396 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 18:34:24.542957 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538400 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538402 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538405 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538408 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538411 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538414 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538417 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538419 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538422 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538424 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538427 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538430 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538432 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538435 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538437 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538439 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538443 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538447 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538450 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 18:34:24.543449 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.538452 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538536 2553 flags.go:64] FLAG: --address="0.0.0.0" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538568 2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538574 2553 flags.go:64] FLAG: --anonymous-auth="true" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538579 2553 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538584 2553 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538587 2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538591 2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538596 2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538599 2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538602 2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538605 2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538608 2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538612 2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538615 2553 flags.go:64] FLAG: --cgroup-root="" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538618 2553 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538621 2553 flags.go:64] FLAG: --client-ca-file="" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538624 2553 flags.go:64] FLAG: --cloud-provider="external" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538627 2553 flags.go:64] FLAG: --cluster-dns="[]" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538631 2553 flags.go:64] FLAG: --cluster-domain="" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538634 2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538637 2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538639 2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538643 2553 flags.go:64] FLAG: --container-log-max-files="5" Sep 08 18:34:24.543907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538647 2553 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538650 2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538652 2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538655 2553 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538658 2553 flags.go:64] FLAG: --contention-profiling="false" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538661 2553 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538664 2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538668 2553 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538671 2553 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538675 2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538678 2553 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538680 2553 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538683 2553 flags.go:64] FLAG: --enable-load-reader="false" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538686 2553 flags.go:64] FLAG: --enable-server="true" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538689 2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538694 2553 flags.go:64] FLAG: --event-burst="100" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538697 2553 flags.go:64] FLAG: --event-qps="50" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538703 2553 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538706 2553 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538709 2553 flags.go:64] FLAG: --eviction-hard="" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538712 2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538716 2553 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538719 2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538722 2553 flags.go:64] FLAG: --eviction-soft="" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538725 2553 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 08 18:34:24.544501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538728 2553 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538731 2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538734 2553 flags.go:64] FLAG: --experimental-mounter-path="" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538737 2553 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538740 2553 flags.go:64] FLAG: --fail-swap-on="true" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538742 2553 flags.go:64] FLAG: --feature-gates="" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538746 2553 flags.go:64] FLAG: --file-check-frequency="20s" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538749 2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538752 2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538755 2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538758 2553 flags.go:64] FLAG: --healthz-port="10248" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538761 2553 flags.go:64] FLAG: --help="false" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538764 2553 flags.go:64] FLAG: --hostname-override="ip-10-0-132-43.ec2.internal" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538767 2553 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538770 2553 flags.go:64] FLAG: --http-check-frequency="20s" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538773 2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538777 2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538780 2553 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538783 2553 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538786 2553 flags.go:64] FLAG: --image-service-endpoint="" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538789 2553 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538793 2553 flags.go:64] FLAG: --kube-api-burst="100" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538796 2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538799 2553 flags.go:64] FLAG: --kube-api-qps="50" Sep 08 18:34:24.545152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538803 2553 flags.go:64] FLAG: --kube-reserved="" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538806 2553 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538809 2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538812 2553 flags.go:64] FLAG: --kubelet-cgroups="" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538815 2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538817 2553 flags.go:64] FLAG: --lock-file="" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538820 2553 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538823 2553 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538826 2553 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538832 2553 flags.go:64] FLAG: --log-json-split-stream="false" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538835 2553 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538837 2553 flags.go:64] FLAG: --log-text-split-stream="false" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538840 2553 flags.go:64] FLAG: --logging-format="text" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538843 2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538847 2553 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538849 2553 flags.go:64] FLAG: --manifest-url="" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538852 2553 flags.go:64] FLAG: --manifest-url-header="" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538856 2553 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538859 2553 flags.go:64] FLAG: --max-open-files="1000000" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538863 2553 flags.go:64] FLAG: --max-pods="110" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538866 2553 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538869 2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538871 2553 flags.go:64] FLAG: --memory-manager-policy="None" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538874 2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538881 2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 08 18:34:24.546095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538884 2553 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538887 2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538894 2553 flags.go:64] FLAG: --node-status-max-images="50" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538897 2553 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538901 2553 flags.go:64] FLAG: --oom-score-adj="-999" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538904 2553 flags.go:64] FLAG: --pod-cidr="" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538907 2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538912 2553 flags.go:64] FLAG: --pod-manifest-path="" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538916 2553 flags.go:64] FLAG: --pod-max-pids="-1" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538919 2553 flags.go:64] FLAG: --pods-per-core="0" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538922 2553 flags.go:64] FLAG: --port="10250" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538925 2553 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538928 2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0312b3f2dbcbd12c7" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538931 2553 flags.go:64] FLAG: --qos-reserved="" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538934 2553 flags.go:64] FLAG: --read-only-port="10255" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538937 2553 flags.go:64] FLAG: --register-node="true" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538940 2553 flags.go:64] FLAG: --register-with-taints="" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538944 2553 flags.go:64] FLAG: --registry-burst="10" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538947 2553 flags.go:64] FLAG: --registry-qps="5" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538949 2553 flags.go:64] FLAG: --reserved-cpus="" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538952 2553 flags.go:64] FLAG: --reserved-memory="" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538956 2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538959 2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538961 2553 flags.go:64] FLAG: --rotate-certificates="false" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538964 2553 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 08 18:34:24.546997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538967 2553 flags.go:64] FLAG: --runonce="false" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538970 2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538973 2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538976 2553 flags.go:64] FLAG: --seccomp-default="false" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538978 2553 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538981 2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538984 2553 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538987 2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538990 2553 flags.go:64] FLAG: --storage-driver-password="root" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538993 2553 flags.go:64] FLAG: --storage-driver-secure="false" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.538996 2553 flags.go:64] FLAG: --storage-driver-table="stats" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539000 2553 flags.go:64] FLAG: --storage-driver-user="root" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539003 2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539006 2553 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539009 2553 flags.go:64] FLAG: --system-cgroups="" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539014 2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539019 2553 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539022 2553 flags.go:64] FLAG: --tls-cert-file="" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539025 2553 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539029 2553 flags.go:64] FLAG: --tls-min-version="" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539032 2553 flags.go:64] FLAG: --tls-private-key-file="" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539035 2553 flags.go:64] FLAG: --topology-manager-policy="none" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539038 2553 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539041 2553 flags.go:64] FLAG: --topology-manager-scope="container" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539044 2553 flags.go:64] FLAG: --v="2" Sep 08 18:34:24.547644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539047 2553 flags.go:64] FLAG: --version="false" Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539052 2553 flags.go:64] FLAG: --vmodule="" Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539055 2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539058 2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539173 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539176 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539179 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539182 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539186 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539188 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539191 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539193 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539196 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539198 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539201 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539204 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539206 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539209 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539212 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539216 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 18:34:24.548249 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539220 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539225 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539231 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539233 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539236 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539239 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539241 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539244 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539247 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539249 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539252 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539255 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539257 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539260 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539263 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539265 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539268 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539270 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539273 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539276 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 18:34:24.548763 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539278 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539281 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539284 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539286 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539289 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539291 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539294 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539296 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539299 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539302 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539306 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539309 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539312 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539314 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539318 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539320 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539323 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539326 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539329 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539331 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 18:34:24.549267 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539334 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539336 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539339 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539341 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539344 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539347 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539349 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539352 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539354 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539357 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539372 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539375 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539377 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539380 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539383 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539386 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539388 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539392 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539395 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 18:34:24.549832 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539398 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539401 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539404 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539408 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539411 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539413 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539416 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539420 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539423 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539425 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539428 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539431 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539433 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539436 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539439 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539441 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539444 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539446 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539449 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539452 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 08 18:34:24.550297 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539454 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 18:34:24.550820 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539457 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 18:34:24.550820 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539459 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 18:34:24.550820 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.539462 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 18:34:24.550820 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.539470 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 18:34:24.550820 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.547718 2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 08 18:34:24.550820 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.547739 2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 08 18:34:24.550820 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.548431 2553 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 08 18:34:24.550820 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.548438 2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 08 18:34:24.550820 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548497 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 18:34:24.550820 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548503 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 18:34:24.550820 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548507 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 18:34:24.550820 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548510 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 18:34:24.550820 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548513 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 18:34:24.550820 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548517 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548522 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548525 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548528 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548531 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548533 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548536 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548539 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548541 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548544 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548547 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548549 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548553 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548557 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548560 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548563 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548566 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548569 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548571 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548575 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 18:34:24.551171 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548578 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548580 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548584 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548586 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548589 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548591 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548594 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548597 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548600 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548603 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548605 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548608 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548611 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548615 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548618 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548621 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548624 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548626 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548629 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 18:34:24.551676 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548632 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548634 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548638 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548641 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548644 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548646 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548649 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548651 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548654 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548656 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548659 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548662 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548664 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548667 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548670 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548672 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548675 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548677 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548680 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548682 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 18:34:24.552137 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548685 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548688 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548690 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548693 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548696 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548698 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548701 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548704 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548707 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548709 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548712 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548714 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548717 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548719 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548721 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548725 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548727 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548730 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548749 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548753 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 18:34:24.552624 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548757 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 18:34:24.553115 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548760 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 18:34:24.553115 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548762 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 18:34:24.553115 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548766 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 18:34:24.553115 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548769 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 18:34:24.553115 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548773 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 18:34:24.553115 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548775 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 18:34:24.553115 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548778 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 18:34:24.553115 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548781 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 18:34:24.553115 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548783 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 18:34:24.553115 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548786 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 18:34:24.553115 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548788 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 18:34:24.553115 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548791 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 18:34:24.553115 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548794 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 18:34:24.553115 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548796 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 18:34:24.553115 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.548801 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548938 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548943 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548946 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548950 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548954 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548958 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548960 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548963 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548966 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548969 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548972 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548975 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548979 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548982 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548984 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548987 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548989 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548992 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548994 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 18:34:24.553500 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.548997 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549000 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549003 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549005 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549008 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549010 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549013 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549015 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549018 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549021 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549023 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549026 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549028 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549031 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549033 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549036 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549038 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549041 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549043 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549046 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 18:34:24.553981 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549048 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549051 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549054 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549056 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549058 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549061 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549065 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549068 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549070 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549073 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549075 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549078 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549080 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549083 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549086 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549088 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549091 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549093 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549096 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 18:34:24.554541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549099 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549101 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549104 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549106 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549109 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549112 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549114 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549117 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549119 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549122 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549125 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549127 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549130 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549133 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549136 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549138 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549141 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549144 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549146 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 18:34:24.554993 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549149 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549152 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549155 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549158 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549160 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549163 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549167 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549170 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549174 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549176 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549179 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549182 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549184 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549187 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549189 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549192 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549195 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549197 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549200 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 18:34:24.555495 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549202 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 18:34:24.555951 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549204 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 18:34:24.555951 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:24.549207 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 18:34:24.555951 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.549212 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 18:34:24.555951 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.549322 2553 server.go:962] "Client rotation is on, will bootstrap in background" Sep 08 18:34:24.555951 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.551464 2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 08 18:34:24.555951 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.552251 2553 server.go:1019] "Starting client certificate rotation" Sep 08 18:34:24.555951 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.552351 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 18:34:24.555951 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.553031 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 18:34:24.555951 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.554601 2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 18:34:24.555951 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.555788 2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 18:34:24.566668 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.566642 2553 log.go:25] "Validated CRI v1 runtime API" Sep 08 18:34:24.569495 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.569480 2553 log.go:25] "Validated CRI v1 image API" Sep 08 18:34:24.570296 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.570279 2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 08 18:34:24.573127 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.573099 2553 fs.go:135] Filesystem UUIDs: map[0006c814-3fa5-4cca-9a33-68b7598e9069:/dev/nvme0n1p4 78782c28-ba8e-4f00-a396-d2588e636a6b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 08 18:34:24.573209 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.573125 2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 08 18:34:24.578762 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.578647 2553 manager.go:216] Machine: {Timestamp:2026-09-08 18:34:24.57816161 +0000 UTC m=+0.284049832 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097979 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ce12caa7e0e4349db45e5176b6c73 SystemUUID:ec2ce12c-aa7e-0e43-49db-45e5176b6c73 BootID:0dcf2de8-c7bb-4c2e-b1f8-f28140ba776f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:49:07:50:63:4b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:49:07:50:63:4b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:57:9b:24:12:3e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 08 18:34:24.578762 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.578751 2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 08 18:34:24.578913 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.578824 2553 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 08 18:34:24.580763 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.580734 2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 08 18:34:24.580898 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.580766 2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-43.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 08 18:34:24.581437 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.581427 2553 topology_manager.go:138] "Creating topology manager with none policy" Sep 08 18:34:24.581470 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.581439 2553 container_manager_linux.go:309] "Creating device plugin manager" Sep 08 18:34:24.581470 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.581447 2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 18:34:24.582642 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.582628 2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 18:34:24.583523 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.583505 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 18:34:24.583905 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.583895 2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 08 18:34:24.587698 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.587685 2553 state_mem.go:36] "Initialized new in-memory state store" Sep 08 18:34:24.587824 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.587814 2553 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 08 18:34:24.587857 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.587845 2553 kubelet.go:486] "Attempting to sync node with API server" Sep 08 18:34:24.587857 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.587856 2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 08 18:34:24.587912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.587871 2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 08 18:34:24.587912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.587880 2553 kubelet.go:398] "Adding apiserver pod source" Sep 08 18:34:24.587912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.587889 2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 08 18:34:24.588233 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.588219 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 18:34:24.588315 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.588238 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 18:34:24.590183 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.590168 2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 08 18:34:24.592507 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.592492 2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 08 18:34:24.592585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.592529 2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 08 18:34:24.593198 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.593185 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 08 18:34:24.593267 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.593206 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 08 18:34:24.593267 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.593215 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 08 18:34:24.593267 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.593226 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 08 18:34:24.593267 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.593238 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 08 18:34:24.593267 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.593247 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 08 18:34:24.593267 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.593266 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 08 18:34:24.593483 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.593274 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 08 18:34:24.593483 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.593284 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 08 18:34:24.593483 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.593293 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 08 18:34:24.593483 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.593306 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 08 18:34:24.593483 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.593316 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 08 18:34:24.594087 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.594074 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 08 18:34:24.594142 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.594088 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 08 18:34:24.597653 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.597637 2553 server.go:1268] "Started kubelet" Sep 08 18:34:24.597764 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.597744 2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-43.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 08 18:34:24.597806 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.597753 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 08 18:34:24.597898 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.597869 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 08 18:34:24.597997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.597891 2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 08 18:34:24.597997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.597956 2553 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 08 18:34:24.598350 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.598311 2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 08 18:34:24.599091 ip-10-0-132-43 systemd[1]: Started Kubernetes Kubelet. Sep 08 18:34:24.599412 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.599394 2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 08 18:34:24.602063 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.602050 2553 server.go:310] "Adding debug handlers to kubelet server" Sep 08 18:34:24.604005 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.603982 2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 08 18:34:24.605584 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.605565 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 08 18:34:24.606046 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.606031 2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 08 18:34:24.606166 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.606146 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Sep 08 18:34:24.606572 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.606553 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d5bf9" Sep 08 18:34:24.606746 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.606728 2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 08 18:34:24.606818 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.606729 2553 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 08 18:34:24.606818 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.606763 2553 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 08 18:34:24.606906 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.606834 2553 reconstruct.go:97] "Volume reconstruction finished" Sep 08 18:34:24.606906 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.606839 2553 reconciler.go:29] "Reconciler: start to sync state" Sep 08 18:34:24.607289 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.607273 2553 factory.go:55] Registering systemd factory Sep 08 18:34:24.607289 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.607290 2553 factory.go:223] Registration of the systemd container factory successfully Sep 08 18:34:24.607519 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.607502 2553 factory.go:153] Registering CRI-O factory Sep 08 18:34:24.607519 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.607520 2553 factory.go:223] Registration of the crio container factory successfully Sep 08 18:34:24.607614 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.607563 2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 08 18:34:24.607614 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.607581 2553 factory.go:103] Registering Raw factory Sep 08 18:34:24.607614 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.607593 2553 manager.go:1196] Started watching for new ooms in manager Sep 08 18:34:24.608001 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.607988 2553 manager.go:349] Starting recovery of all containers Sep 08 18:34:24.610540 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.610516 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 08 18:34:24.610921 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.610902 2553 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 08 18:34:24.611533 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.610658 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-43.ec2.internal.18d36c37f5b6cab4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-43.ec2.internal,UID:ip-10-0-132-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-43.ec2.internal,},FirstTimestamp:2026-09-08 18:34:24.597617332 +0000 UTC m=+0.303505555,LastTimestamp:2026-09-08 18:34:24.597617332 +0000 UTC m=+0.303505555,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-43.ec2.internal,}" Sep 08 18:34:24.614494 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.613985 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d5bf9" Sep 08 18:34:24.618486 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.618326 2553 manager.go:354] Recovery completed Sep 08 18:34:24.623425 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.623406 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 18:34:24.625497 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.625477 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientMemory" Sep 08 18:34:24.625557 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.625513 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 18:34:24.625557 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.625526 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientPID" Sep 08 18:34:24.626131 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.626116 2553 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 08 18:34:24.626131 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.626129 2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 08 18:34:24.626237 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.626146 2553 state_mem.go:36] "Initialized new in-memory state store" Sep 08 18:34:24.626237 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.626165 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 18:34:24.626988 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.626970 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientMemory" Sep 08 18:34:24.627047 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.627003 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 18:34:24.627047 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.627019 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientPID" Sep 08 18:34:24.627523 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.627458 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-43.ec2.internal.18d36c37f76032dd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-43.ec2.internal,UID:ip-10-0-132-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-43.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-43.ec2.internal,},FirstTimestamp:2026-09-08 18:34:24.625496797 +0000 UTC m=+0.331385029,LastTimestamp:2026-09-08 18:34:24.625496797 +0000 UTC m=+0.331385029,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-43.ec2.internal,}" Sep 08 18:34:24.628238 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.628225 2553 policy_none.go:49] "None policy: Start" Sep 08 18:34:24.628285 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.628241 2553 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 08 18:34:24.628285 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.628252 2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 08 18:34:24.630260 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.630248 2553 policy_none.go:47] "Start" Sep 08 18:34:24.655174 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.655157 2553 manager.go:337] "Starting Device Plugin manager" Sep 08 18:34:24.655279 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.655200 2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 08 18:34:24.655279 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.655212 2553 server.go:87] "Starting device plugin registration server" Sep 08 18:34:24.655607 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.655593 2553 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 08 18:34:24.655697 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.655610 2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 08 18:34:24.655697 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.655680 2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 08 18:34:24.655805 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.655754 2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 08 18:34:24.655805 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.655763 2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 08 18:34:24.656546 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.656524 2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 08 18:34:24.656629 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.656582 2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-43.ec2.internal\" not found" Sep 08 18:34:24.737097 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.737059 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 08 18:34:24.739265 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.738271 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 08 18:34:24.739265 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.738315 2553 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 08 18:34:24.739265 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.738339 2553 kubelet.go:2448] "Starting kubelet main sync loop" Sep 08 18:34:24.739265 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.738438 2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 08 18:34:24.740154 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.740135 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 18:34:24.756255 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.756239 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 18:34:24.757032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.757016 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientMemory" Sep 08 18:34:24.757095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.757046 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 18:34:24.757095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.757056 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientPID" Sep 08 18:34:24.757095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.757076 2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-43.ec2.internal" Sep 08 18:34:24.764388 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.764374 2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-43.ec2.internal" Sep 08 18:34:24.764438 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.764395 2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-43.ec2.internal\": node \"ip-10-0-132-43.ec2.internal\" not found" Sep 08 18:34:24.780755 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.780729 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Sep 08 18:34:24.838974 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.838884 2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal"] Sep 08 18:34:24.838974 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.838974 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 18:34:24.840474 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.840451 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientMemory" Sep 08 18:34:24.840474 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.840477 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 18:34:24.840605 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.840486 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientPID" Sep 08 18:34:24.842787 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.842775 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 18:34:24.842929 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.842911 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" Sep 08 18:34:24.842981 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.842942 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 18:34:24.843521 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.843504 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientMemory" Sep 08 18:34:24.843606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.843504 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientMemory" Sep 08 18:34:24.843606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.843541 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 18:34:24.843606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.843556 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientPID" Sep 08 18:34:24.843606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.843557 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 18:34:24.843782 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.843630 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientPID" Sep 08 18:34:24.845842 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.845825 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Sep 08 18:34:24.845928 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.845854 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 18:34:24.846509 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.846496 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientMemory" Sep 08 18:34:24.846589 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.846524 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 18:34:24.846589 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.846539 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeHasSufficientPID" Sep 08 18:34:24.876009 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.875986 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-43.ec2.internal\" not found" node="ip-10-0-132-43.ec2.internal" Sep 08 18:34:24.880444 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.880427 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-43.ec2.internal\" not found" node="ip-10-0-132-43.ec2.internal" Sep 08 18:34:24.881411 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.881397 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Sep 08 18:34:24.908458 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.908434 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c0a1ff402e4c440e263b451c0fbfb11-config\") pod \"kube-apiserver-proxy-ip-10-0-132-43.ec2.internal\" (UID: \"1c0a1ff402e4c440e263b451c0fbfb11\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" Sep 08 18:34:24.908540 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.908462 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8737f89cceb63fd2f10ea539aa22dbf1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal\" (UID: \"8737f89cceb63fd2f10ea539aa22dbf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Sep 08 18:34:24.908540 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:24.908488 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8737f89cceb63fd2f10ea539aa22dbf1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal\" (UID: \"8737f89cceb63fd2f10ea539aa22dbf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Sep 08 18:34:24.981824 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:24.981799 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Sep 08 18:34:25.009134 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.009115 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c0a1ff402e4c440e263b451c0fbfb11-config\") pod \"kube-apiserver-proxy-ip-10-0-132-43.ec2.internal\" (UID: \"1c0a1ff402e4c440e263b451c0fbfb11\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" Sep 08 18:34:25.009209 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.009139 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8737f89cceb63fd2f10ea539aa22dbf1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal\" (UID: \"8737f89cceb63fd2f10ea539aa22dbf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Sep 08 18:34:25.009209 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.009155 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8737f89cceb63fd2f10ea539aa22dbf1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal\" (UID: \"8737f89cceb63fd2f10ea539aa22dbf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Sep 08 18:34:25.009303 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.009226 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8737f89cceb63fd2f10ea539aa22dbf1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal\" (UID: \"8737f89cceb63fd2f10ea539aa22dbf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Sep 08 18:34:25.009303 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.009230 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1c0a1ff402e4c440e263b451c0fbfb11-config\") pod \"kube-apiserver-proxy-ip-10-0-132-43.ec2.internal\" (UID: \"1c0a1ff402e4c440e263b451c0fbfb11\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" Sep 08 18:34:25.009303 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.009224 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8737f89cceb63fd2f10ea539aa22dbf1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal\" (UID: \"8737f89cceb63fd2f10ea539aa22dbf1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Sep 08 18:34:25.082433 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:25.082394 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Sep 08 18:34:25.179140 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.179075 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" Sep 08 18:34:25.182514 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:25.182496 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Sep 08 18:34:25.183696 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.183684 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Sep 08 18:34:25.283264 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:25.283235 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Sep 08 18:34:25.383813 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:25.383793 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Sep 08 18:34:25.484166 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:25.484107 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Sep 08 18:34:25.555337 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.555308 2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 08 18:34:25.555810 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.555488 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 18:34:25.585024 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:25.584989 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Sep 08 18:34:25.618192 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.618158 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 18:29:25 +0000 UTC" deadline="2028-05-16 08:30:25.526567522 +0000 UTC" Sep 08 18:34:25.618192 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.618187 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14773h55m59.908383519s" Sep 08 18:34:25.686140 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:25.686108 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-43.ec2.internal\" not found" Sep 08 18:34:25.696476 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.696454 2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 18:34:25.755370 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.755310 2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 18:34:25.806547 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.806518 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" Sep 08 18:34:25.818005 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.817978 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 18:34:25.819685 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.819668 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" Sep 08 18:34:25.828005 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.827984 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 18:34:25.875774 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:25.875745 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0a1ff402e4c440e263b451c0fbfb11.slice/crio-584f029d7c0d431f9f103a382f1f3991f7e11bd7e266c5e1569a43fdda94c391 WatchSource:0}: Error finding container 584f029d7c0d431f9f103a382f1f3991f7e11bd7e266c5e1569a43fdda94c391: Status 404 returned error can't find the container with id 584f029d7c0d431f9f103a382f1f3991f7e11bd7e266c5e1569a43fdda94c391 Sep 08 18:34:25.876180 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:25.876160 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8737f89cceb63fd2f10ea539aa22dbf1.slice/crio-74f5f88188776a0100459de5ff80d4220041a68f3b80e593c405134da1f8a12a WatchSource:0}: Error finding container 74f5f88188776a0100459de5ff80d4220041a68f3b80e593c405134da1f8a12a: Status 404 returned error can't find the container with id 74f5f88188776a0100459de5ff80d4220041a68f3b80e593c405134da1f8a12a Sep 08 18:34:25.881030 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:25.881011 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 18:34:26.102139 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.102058 2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 18:34:26.450044 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.449951 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 18:34:26.588915 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.588877 2553 apiserver.go:52] "Watching apiserver" Sep 08 18:34:26.604334 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.604307 2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 08 18:34:26.605461 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.605415 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-f4hs5","kube-system/global-pull-secret-syncer-tzbd8","openshift-cluster-node-tuning-operator/tuned-klsxq","openshift-multus/multus-7pnc2","openshift-multus/multus-additional-cni-plugins-gchnh","openshift-multus/network-metrics-daemon-sjrbw","openshift-network-diagnostics/network-check-target-bwbtk","kube-system/konnectivity-agent-j9kxx","kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv","openshift-dns/node-resolver-xmpj8","openshift-image-registry/node-ca-jq7qc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal","openshift-network-operator/iptables-alerter-xpb2v"] Sep 08 18:34:26.606430 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.606401 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 08 18:34:26.608884 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.608488 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:26.608884 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:26.608568 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:26.611128 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.610386 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xmpj8" Sep 08 18:34:26.614478 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.614037 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.616133 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.615129 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.616133 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.615979 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x59sx\"" Sep 08 18:34:26.616288 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.616252 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 08 18:34:26.616380 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.616351 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 08 18:34:26.617074 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.616662 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 08 18:34:26.617074 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.616938 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m7wkk\"" Sep 08 18:34:26.618006 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.617985 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.619993 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.619973 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:26.620090 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:26.620064 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:26.621177 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.620344 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqp9\" (UniqueName: \"kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9\") pod \"network-check-target-bwbtk\" (UID: \"b5899806-e48c-4bdf-a215-12a2e468f694\") " pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:26.622939 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.621960 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 08 18:34:26.622939 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.621956 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 08 18:34:26.622939 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.622005 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 08 18:34:26.622939 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.622201 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 08 18:34:26.622939 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.622220 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 08 18:34:26.622939 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.622372 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 08 18:34:26.622939 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.622410 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 18:34:26.622939 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.622428 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4wgz8\"" Sep 08 18:34:26.622939 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.622562 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 08 18:34:26.622939 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.622602 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xpb2v" Sep 08 18:34:26.622939 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.622709 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6tr75\"" Sep 08 18:34:26.625851 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.624887 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j9kxx" Sep 08 18:34:26.625851 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.625260 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 08 18:34:26.625851 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.625293 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8vp94\"" Sep 08 18:34:26.625851 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.625650 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 08 18:34:26.626117 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.625858 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 08 18:34:26.627843 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.627826 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:26.627932 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:26.627907 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:26.627932 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.627918 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 08 18:34:26.628040 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.627969 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2c64l\"" Sep 08 18:34:26.628094 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.628039 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 08 18:34:26.630084 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.630036 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.632757 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.632736 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-snghn\"" Sep 08 18:34:26.633016 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.632996 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.633185 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.633162 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 08 18:34:26.633517 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.633498 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 08 18:34:26.633771 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.633753 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 08 18:34:26.635202 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.635181 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jq7qc" Sep 08 18:34:26.635945 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.635926 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 08 18:34:26.636208 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.636188 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 08 18:34:26.638361 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.636388 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 08 18:34:26.638361 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.636673 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 08 18:34:26.638361 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.636884 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-44njf\"" Sep 08 18:34:26.638361 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.637479 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 08 18:34:26.638361 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.637688 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 08 18:34:26.638361 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.638219 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 08 18:34:26.638749 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.638495 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 08 18:34:26.638749 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.638698 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 08 18:34:26.640354 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.639330 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5tdkq\"" Sep 08 18:34:26.641480 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.641390 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dj2jq" Sep 08 18:34:26.652532 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.652511 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dj2jq" Sep 08 18:34:26.708071 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.708048 2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 08 18:34:26.721430 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721405 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-registration-dir\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.721561 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721447 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-cnibin\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.721561 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721476 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-cni-binary-copy\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.721561 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721504 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b33d5d-d099-464a-adcc-03562ae049c0-tmp-dir\") pod \"node-resolver-xmpj8\" (UID: \"96b33d5d-d099-464a-adcc-03562ae049c0\") " pod="openshift-dns/node-resolver-xmpj8" Sep 08 18:34:26.721561 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721529 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-tuned\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.721561 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721552 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-sys-fs\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.721843 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721576 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-systemd-units\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.721843 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721696 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-var-lib-cni-bin\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.721843 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721727 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82-iptables-alerter-script\") pod \"iptables-alerter-xpb2v\" (UID: \"df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82\") " pod="openshift-network-operator/iptables-alerter-xpb2v" Sep 08 18:34:26.721843 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721765 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7aa9c205-ae85-4940-ba9a-b9ba08937086-agent-certs\") pod \"konnectivity-agent-j9kxx\" (UID: \"7aa9c205-ae85-4940-ba9a-b9ba08937086\") " pod="kube-system/konnectivity-agent-j9kxx" Sep 08 18:34:26.721843 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721794 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.721843 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721837 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-sysctl-conf\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.722181 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721883 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-var-lib-kubelet\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.722181 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721907 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa2faa21-f7f1-4010-851b-bd9244d70ea5-multus-daemon-config\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.722181 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721933 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-multus-socket-dir-parent\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.722181 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.721982 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-hostroot\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.722181 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722008 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20d4c356-520e-4dd1-b36d-40197469dd3c-env-overrides\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.722181 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722033 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20d4c356-520e-4dd1-b36d-40197469dd3c-ovn-node-metrics-cert\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.722181 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722059 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbmqc\" (UniqueName: \"kubernetes.io/projected/1911e404-7356-4fd2-86af-acc9012c5a4b-kube-api-access-dbmqc\") pod \"node-ca-jq7qc\" (UID: \"1911e404-7356-4fd2-86af-acc9012c5a4b\") " pod="openshift-image-registry/node-ca-jq7qc" Sep 08 18:34:26.722181 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722082 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-tmp\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.722181 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722115 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-var-lib-cni-multus\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.722181 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722182 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxjbb\" (UniqueName: \"kubernetes.io/projected/df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82-kube-api-access-bxjbb\") pod \"iptables-alerter-xpb2v\" (UID: \"df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82\") " pod="openshift-network-operator/iptables-alerter-xpb2v" Sep 08 18:34:26.722693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722208 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-slash\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.722693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722235 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1911e404-7356-4fd2-86af-acc9012c5a4b-serviceca\") pod \"node-ca-jq7qc\" (UID: \"1911e404-7356-4fd2-86af-acc9012c5a4b\") " pod="openshift-image-registry/node-ca-jq7qc" Sep 08 18:34:26.722693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722261 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/96b33d5d-d099-464a-adcc-03562ae049c0-hosts-file\") pod \"node-resolver-xmpj8\" (UID: \"96b33d5d-d099-464a-adcc-03562ae049c0\") " pod="openshift-dns/node-resolver-xmpj8" Sep 08 18:34:26.722693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722284 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-run-netns\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.722693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722308 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-run-ovn\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.722693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722330 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-sysctl-d\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.722693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722366 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqp9\" (UniqueName: \"kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9\") pod \"network-check-target-bwbtk\" (UID: \"b5899806-e48c-4bdf-a215-12a2e468f694\") " pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:26.722693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722396 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-multus-conf-dir\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.722693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722459 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-socket-dir\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.722693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722577 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-etc-selinux\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.722693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722608 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82gg8\" (UniqueName: \"kubernetes.io/projected/effd0529-39cd-43ca-96d5-6fde24d932db-kube-api-access-82gg8\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.722693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722649 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-cni-bin\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.722693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722674 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-run-multus-certs\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.722693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722700 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722736 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-run-systemd\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722778 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-var-lib-openvswitch\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722820 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-etc-openvswitch\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722855 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-node-log\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722882 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-log-socket\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722920 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20d4c356-520e-4dd1-b36d-40197469dd3c-ovnkube-config\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722950 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvwjx\" (UniqueName: \"kubernetes.io/projected/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-kube-api-access-rvwjx\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.722989 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-systemd\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723019 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-kubernetes\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723044 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-var-lib-kubelet\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723065 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-host\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723087 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-cnibin\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723114 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-system-cni-dir\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723131 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-kubelet\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723145 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-system-cni-dir\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.723417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723168 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723197 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8hjj\" (UniqueName: \"kubernetes.io/projected/40b6e42c-6d96-445d-8b83-108320730adb-kube-api-access-q8hjj\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723221 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-run-openvswitch\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723239 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1911e404-7356-4fd2-86af-acc9012c5a4b-host\") pod \"node-ca-jq7qc\" (UID: \"1911e404-7356-4fd2-86af-acc9012c5a4b\") " pod="openshift-image-registry/node-ca-jq7qc" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723263 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-modprobe-d\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723308 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-sysconfig\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723330 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-etc-kubernetes\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723355 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7aa9c205-ae85-4940-ba9a-b9ba08937086-konnectivity-ca\") pod \"konnectivity-agent-j9kxx\" (UID: \"7aa9c205-ae85-4940-ba9a-b9ba08937086\") " pod="kube-system/konnectivity-agent-j9kxx" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723394 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-kubelet-config\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723424 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20d4c356-520e-4dd1-b36d-40197469dd3c-ovnkube-script-lib\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723450 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4rlw\" (UniqueName: \"kubernetes.io/projected/20d4c356-520e-4dd1-b36d-40197469dd3c-kube-api-access-c4rlw\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723481 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723509 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-cni-netd\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723533 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msh4j\" (UniqueName: \"kubernetes.io/projected/96b33d5d-d099-464a-adcc-03562ae049c0-kube-api-access-msh4j\") pod \"node-resolver-xmpj8\" (UID: \"96b33d5d-d099-464a-adcc-03562ae049c0\") " pod="openshift-dns/node-resolver-xmpj8" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723563 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgt7r\" (UniqueName: \"kubernetes.io/projected/aa2faa21-f7f1-4010-851b-bd9244d70ea5-kube-api-access-jgt7r\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723586 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-device-dir\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.724211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723609 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-dbus\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:26.725078 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723640 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa2faa21-f7f1-4010-851b-bd9244d70ea5-cni-binary-copy\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.725078 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723666 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-sys\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.725078 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723700 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-lib-modules\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.725078 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723724 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-multus-cni-dir\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.725078 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723745 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82-host-slash\") pod \"iptables-alerter-xpb2v\" (UID: \"df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82\") " pod="openshift-network-operator/iptables-alerter-xpb2v" Sep 08 18:34:26.725078 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723762 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.725078 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723790 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-run-ovn-kubernetes\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.725078 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723814 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-run\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.725078 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723847 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ckc8\" (UniqueName: \"kubernetes.io/projected/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-kube-api-access-5ckc8\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.725078 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723876 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:26.725078 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723916 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.725078 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723937 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-run-k8s-cni-cncf-io\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.725078 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.723961 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-run-netns\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.728638 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:26.728615 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 18:34:26.728733 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:26.728648 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 18:34:26.728733 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:26.728663 2553 projected.go:196] Error preparing data for projected volume kube-api-access-bgqp9 for pod openshift-network-diagnostics/network-check-target-bwbtk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:26.728860 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:26.728737 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9 podName:b5899806-e48c-4bdf-a215-12a2e468f694 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:27.228705277 +0000 UTC m=+2.934593511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bgqp9" (UniqueName: "kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9") pod "network-check-target-bwbtk" (UID: "b5899806-e48c-4bdf-a215-12a2e468f694") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:26.746017 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.745967 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" event={"ID":"8737f89cceb63fd2f10ea539aa22dbf1","Type":"ContainerStarted","Data":"74f5f88188776a0100459de5ff80d4220041a68f3b80e593c405134da1f8a12a"} Sep 08 18:34:26.749783 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.749755 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" event={"ID":"1c0a1ff402e4c440e263b451c0fbfb11","Type":"ContainerStarted","Data":"584f029d7c0d431f9f103a382f1f3991f7e11bd7e266c5e1569a43fdda94c391"} Sep 08 18:34:26.824615 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.824576 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-run-netns\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.824800 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.824623 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-registration-dir\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.824800 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.824649 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-cnibin\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.824800 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.824683 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-run-netns\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.824800 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.824672 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-cni-binary-copy\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.824800 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.824731 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b33d5d-d099-464a-adcc-03562ae049c0-tmp-dir\") pod \"node-resolver-xmpj8\" (UID: \"96b33d5d-d099-464a-adcc-03562ae049c0\") " pod="openshift-dns/node-resolver-xmpj8" Sep 08 18:34:26.824800 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.824744 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-registration-dir\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.824800 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.824755 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-tuned\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.824806 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-sys-fs\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.824870 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-systemd-units\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.824901 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-var-lib-cni-bin\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.824928 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82-iptables-alerter-script\") pod \"iptables-alerter-xpb2v\" (UID: \"df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82\") " pod="openshift-network-operator/iptables-alerter-xpb2v" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.824953 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7aa9c205-ae85-4940-ba9a-b9ba08937086-agent-certs\") pod \"konnectivity-agent-j9kxx\" (UID: \"7aa9c205-ae85-4940-ba9a-b9ba08937086\") " pod="kube-system/konnectivity-agent-j9kxx" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.824977 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825005 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-sysctl-conf\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825023 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-cnibin\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825053 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b33d5d-d099-464a-adcc-03562ae049c0-tmp-dir\") pod \"node-resolver-xmpj8\" (UID: \"96b33d5d-d099-464a-adcc-03562ae049c0\") " pod="openshift-dns/node-resolver-xmpj8" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825031 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-var-lib-kubelet\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825070 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-var-lib-kubelet\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825091 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa2faa21-f7f1-4010-851b-bd9244d70ea5-multus-daemon-config\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825120 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-multus-socket-dir-parent\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825143 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-hostroot\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825172 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20d4c356-520e-4dd1-b36d-40197469dd3c-env-overrides\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825195 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20d4c356-520e-4dd1-b36d-40197469dd3c-ovn-node-metrics-cert\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.825229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825220 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbmqc\" (UniqueName: \"kubernetes.io/projected/1911e404-7356-4fd2-86af-acc9012c5a4b-kube-api-access-dbmqc\") pod \"node-ca-jq7qc\" (UID: \"1911e404-7356-4fd2-86af-acc9012c5a4b\") " pod="openshift-image-registry/node-ca-jq7qc" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825242 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-tmp\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825268 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-var-lib-cni-multus\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825292 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxjbb\" (UniqueName: \"kubernetes.io/projected/df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82-kube-api-access-bxjbb\") pod \"iptables-alerter-xpb2v\" (UID: \"df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82\") " pod="openshift-network-operator/iptables-alerter-xpb2v" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825317 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-slash\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825342 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1911e404-7356-4fd2-86af-acc9012c5a4b-serviceca\") pod \"node-ca-jq7qc\" (UID: \"1911e404-7356-4fd2-86af-acc9012c5a4b\") " pod="openshift-image-registry/node-ca-jq7qc" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825368 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/96b33d5d-d099-464a-adcc-03562ae049c0-hosts-file\") pod \"node-resolver-xmpj8\" (UID: \"96b33d5d-d099-464a-adcc-03562ae049c0\") " pod="openshift-dns/node-resolver-xmpj8" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825415 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-run-netns\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825442 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-run-ovn\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825465 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-sysctl-d\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825506 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-multus-conf-dir\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825532 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-socket-dir\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825557 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-etc-selinux\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825583 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82gg8\" (UniqueName: \"kubernetes.io/projected/effd0529-39cd-43ca-96d5-6fde24d932db-kube-api-access-82gg8\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825618 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-cni-bin\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825643 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-run-multus-certs\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825670 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.826056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825695 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-run-systemd\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825722 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-var-lib-openvswitch\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825755 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-etc-openvswitch\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825774 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-sys-fs\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825783 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-node-log\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825783 2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825813 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-log-socket\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825839 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20d4c356-520e-4dd1-b36d-40197469dd3c-ovnkube-config\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825832 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-var-lib-cni-bin\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825869 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwjx\" (UniqueName: \"kubernetes.io/projected/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-kube-api-access-rvwjx\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825889 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-systemd-units\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825902 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-systemd\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825939 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825957 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-kubernetes\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825986 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-var-lib-kubelet\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826010 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-host\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826033 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-cnibin\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826059 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-system-cni-dir\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.826912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826084 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-kubelet\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826110 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-system-cni-dir\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826137 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826168 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8hjj\" (UniqueName: \"kubernetes.io/projected/40b6e42c-6d96-445d-8b83-108320730adb-kube-api-access-q8hjj\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826195 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-run-openvswitch\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826221 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1911e404-7356-4fd2-86af-acc9012c5a4b-host\") pod \"node-ca-jq7qc\" (UID: \"1911e404-7356-4fd2-86af-acc9012c5a4b\") " pod="openshift-image-registry/node-ca-jq7qc" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826249 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-modprobe-d\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826273 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-sysconfig\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826298 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-etc-kubernetes\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826334 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7aa9c205-ae85-4940-ba9a-b9ba08937086-konnectivity-ca\") pod \"konnectivity-agent-j9kxx\" (UID: \"7aa9c205-ae85-4940-ba9a-b9ba08937086\") " pod="kube-system/konnectivity-agent-j9kxx" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826368 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-kubelet-config\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826399 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20d4c356-520e-4dd1-b36d-40197469dd3c-ovnkube-script-lib\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826434 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4rlw\" (UniqueName: \"kubernetes.io/projected/20d4c356-520e-4dd1-b36d-40197469dd3c-kube-api-access-c4rlw\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826480 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826512 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-cni-netd\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826546 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msh4j\" (UniqueName: \"kubernetes.io/projected/96b33d5d-d099-464a-adcc-03562ae049c0-kube-api-access-msh4j\") pod \"node-resolver-xmpj8\" (UID: \"96b33d5d-d099-464a-adcc-03562ae049c0\") " pod="openshift-dns/node-resolver-xmpj8" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826578 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgt7r\" (UniqueName: \"kubernetes.io/projected/aa2faa21-f7f1-4010-851b-bd9244d70ea5-kube-api-access-jgt7r\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.827850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826610 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-device-dir\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826644 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-dbus\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826652 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-cni-binary-copy\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826675 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa2faa21-f7f1-4010-851b-bd9244d70ea5-cni-binary-copy\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826706 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-sys\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826737 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-lib-modules\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826770 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-multus-cni-dir\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826802 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82-host-slash\") pod \"iptables-alerter-xpb2v\" (UID: \"df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82\") " pod="openshift-network-operator/iptables-alerter-xpb2v" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826818 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-sysctl-conf\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826829 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826861 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-run-ovn-kubernetes\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826891 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-run\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826919 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ckc8\" (UniqueName: \"kubernetes.io/projected/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-kube-api-access-5ckc8\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826931 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa2faa21-f7f1-4010-851b-bd9244d70ea5-multus-daemon-config\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826946 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.826993 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827010 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-run-openvswitch\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.828724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827034 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-run-k8s-cni-cncf-io\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827059 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1911e404-7356-4fd2-86af-acc9012c5a4b-host\") pod \"node-ca-jq7qc\" (UID: \"1911e404-7356-4fd2-86af-acc9012c5a4b\") " pod="openshift-image-registry/node-ca-jq7qc" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827093 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-cni-bin\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827157 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-run-systemd\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827186 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-multus-socket-dir-parent\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827199 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-var-lib-openvswitch\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.825721 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82-iptables-alerter-script\") pod \"iptables-alerter-xpb2v\" (UID: \"df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82\") " pod="openshift-network-operator/iptables-alerter-xpb2v" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827229 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-cni-netd\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827252 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-etc-openvswitch\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827270 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-hostroot\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827295 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-node-log\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827340 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-log-socket\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827399 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-sys\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827519 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-lib-modules\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827581 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-multus-cni-dir\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827622 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82-host-slash\") pod \"iptables-alerter-xpb2v\" (UID: \"df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82\") " pod="openshift-network-operator/iptables-alerter-xpb2v" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.827738 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20d4c356-520e-4dd1-b36d-40197469dd3c-env-overrides\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.828069 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.829606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.828162 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-systemd\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.828254 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-kubernetes\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.828298 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-run-ovn-kubernetes\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.828310 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-var-lib-kubelet\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.828353 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-run\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.828374 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-host\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.828409 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-cnibin\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.828474 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-system-cni-dir\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.828539 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-system-cni-dir\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:26.828613 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.828635 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-device-dir\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:26.828672 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret podName:f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c nodeName:}" failed. No retries permitted until 2026-09-08 18:34:27.328654395 +0000 UTC m=+3.034542622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret") pod "global-pull-secret-syncer-tzbd8" (UID: "f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c") : object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.828694 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-dbus\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:26.828791 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.828858 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.828944 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.828985 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-run-k8s-cni-cncf-io\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829045 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-kubelet\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.831032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829060 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-modprobe-d\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829090 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-sysconfig\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829097 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-run-ovn\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829126 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-etc-kubernetes\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829132 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-slash\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829156 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa2faa21-f7f1-4010-851b-bd9244d70ea5-cni-binary-copy\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829210 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-var-lib-cni-multus\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829412 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/96b33d5d-d099-464a-adcc-03562ae049c0-hosts-file\") pod \"node-resolver-xmpj8\" (UID: \"96b33d5d-d099-464a-adcc-03562ae049c0\") " pod="openshift-dns/node-resolver-xmpj8" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829472 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20d4c356-520e-4dd1-b36d-40197469dd3c-host-run-netns\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829545 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-etc-selinux\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829644 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/effd0529-39cd-43ca-96d5-6fde24d932db-socket-dir\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829695 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7aa9c205-ae85-4940-ba9a-b9ba08937086-konnectivity-ca\") pod \"konnectivity-agent-j9kxx\" (UID: \"7aa9c205-ae85-4940-ba9a-b9ba08937086\") " pod="kube-system/konnectivity-agent-j9kxx" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829724 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-sysctl-d\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829772 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-kubelet-config\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829779 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-multus-conf-dir\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.829829 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:26.830032 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs podName:40b6e42c-6d96-445d-8b83-108320730adb nodeName:}" failed. No retries permitted until 2026-09-08 18:34:27.330013456 +0000 UTC m=+3.035901685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs") pod "network-metrics-daemon-sjrbw" (UID: "40b6e42c-6d96-445d-8b83-108320730adb") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:26.832065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.830070 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1911e404-7356-4fd2-86af-acc9012c5a4b-serviceca\") pod \"node-ca-jq7qc\" (UID: \"1911e404-7356-4fd2-86af-acc9012c5a4b\") " pod="openshift-image-registry/node-ca-jq7qc" Sep 08 18:34:26.832691 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.830399 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa2faa21-f7f1-4010-851b-bd9244d70ea5-host-run-multus-certs\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.832691 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.830659 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7aa9c205-ae85-4940-ba9a-b9ba08937086-agent-certs\") pod \"konnectivity-agent-j9kxx\" (UID: \"7aa9c205-ae85-4940-ba9a-b9ba08937086\") " pod="kube-system/konnectivity-agent-j9kxx" Sep 08 18:34:26.832691 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.830699 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20d4c356-520e-4dd1-b36d-40197469dd3c-ovnkube-config\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.832691 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.830718 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-etc-tuned\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.832691 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.830929 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20d4c356-520e-4dd1-b36d-40197469dd3c-ovn-node-metrics-cert\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.832691 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.831580 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20d4c356-520e-4dd1-b36d-40197469dd3c-ovnkube-script-lib\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.832691 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.832599 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-tmp\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.836416 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.835429 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82gg8\" (UniqueName: \"kubernetes.io/projected/effd0529-39cd-43ca-96d5-6fde24d932db-kube-api-access-82gg8\") pod \"aws-ebs-csi-driver-node-l9kqv\" (UID: \"effd0529-39cd-43ca-96d5-6fde24d932db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.840888 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.838602 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8hjj\" (UniqueName: \"kubernetes.io/projected/40b6e42c-6d96-445d-8b83-108320730adb-kube-api-access-q8hjj\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:26.840888 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.839214 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msh4j\" (UniqueName: \"kubernetes.io/projected/96b33d5d-d099-464a-adcc-03562ae049c0-kube-api-access-msh4j\") pod \"node-resolver-xmpj8\" (UID: \"96b33d5d-d099-464a-adcc-03562ae049c0\") " pod="openshift-dns/node-resolver-xmpj8" Sep 08 18:34:26.842330 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.842292 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgt7r\" (UniqueName: \"kubernetes.io/projected/aa2faa21-f7f1-4010-851b-bd9244d70ea5-kube-api-access-jgt7r\") pod \"multus-7pnc2\" (UID: \"aa2faa21-f7f1-4010-851b-bd9244d70ea5\") " pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.844481 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.843251 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwjx\" (UniqueName: \"kubernetes.io/projected/d8e54f1e-5ce8-481f-a6cb-c01a98251ed7-kube-api-access-rvwjx\") pod \"multus-additional-cni-plugins-gchnh\" (UID: \"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7\") " pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.844481 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.844095 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxjbb\" (UniqueName: \"kubernetes.io/projected/df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82-kube-api-access-bxjbb\") pod \"iptables-alerter-xpb2v\" (UID: \"df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82\") " pod="openshift-network-operator/iptables-alerter-xpb2v" Sep 08 18:34:26.844481 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.844211 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbmqc\" (UniqueName: \"kubernetes.io/projected/1911e404-7356-4fd2-86af-acc9012c5a4b-kube-api-access-dbmqc\") pod \"node-ca-jq7qc\" (UID: \"1911e404-7356-4fd2-86af-acc9012c5a4b\") " pod="openshift-image-registry/node-ca-jq7qc" Sep 08 18:34:26.846434 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.846410 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4rlw\" (UniqueName: \"kubernetes.io/projected/20d4c356-520e-4dd1-b36d-40197469dd3c-kube-api-access-c4rlw\") pod \"ovnkube-node-f4hs5\" (UID: \"20d4c356-520e-4dd1-b36d-40197469dd3c\") " pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.846880 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.846852 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ckc8\" (UniqueName: \"kubernetes.io/projected/5ba2ecc9-ebf7-4073-849d-c1700b3cecfb-kube-api-access-5ckc8\") pod \"tuned-klsxq\" (UID: \"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb\") " pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.928178 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.928145 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xmpj8" Sep 08 18:34:26.939166 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:26.939123 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b33d5d_d099_464a_adcc_03562ae049c0.slice/crio-1d06911d1448e87bbe7bb1c23fabacb475a8fda26301b2b1958b8716a3da41be WatchSource:0}: Error finding container 1d06911d1448e87bbe7bb1c23fabacb475a8fda26301b2b1958b8716a3da41be: Status 404 returned error can't find the container with id 1d06911d1448e87bbe7bb1c23fabacb475a8fda26301b2b1958b8716a3da41be Sep 08 18:34:26.940693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.940668 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-klsxq" Sep 08 18:34:26.950802 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.950776 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7pnc2" Sep 08 18:34:26.951670 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:26.951610 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba2ecc9_ebf7_4073_849d_c1700b3cecfb.slice/crio-e5465c12f1d483cd063d5ad520c89c3b30f7980d292b7446edc5bff258c42cd9 WatchSource:0}: Error finding container e5465c12f1d483cd063d5ad520c89c3b30f7980d292b7446edc5bff258c42cd9: Status 404 returned error can't find the container with id e5465c12f1d483cd063d5ad520c89c3b30f7980d292b7446edc5bff258c42cd9 Sep 08 18:34:26.959372 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.959344 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gchnh" Sep 08 18:34:26.961955 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:26.961915 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2faa21_f7f1_4010_851b_bd9244d70ea5.slice/crio-71dee676833decdd0c1ad6dae9c50ab78205c57105150c8f5b6278364731251d WatchSource:0}: Error finding container 71dee676833decdd0c1ad6dae9c50ab78205c57105150c8f5b6278364731251d: Status 404 returned error can't find the container with id 71dee676833decdd0c1ad6dae9c50ab78205c57105150c8f5b6278364731251d Sep 08 18:34:26.969732 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.969709 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xpb2v" Sep 08 18:34:26.970021 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:26.969973 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e54f1e_5ce8_481f_a6cb_c01a98251ed7.slice/crio-28a533c26f06088c258ed8d21aef048d240e566a6e224ebfe9e336d879905fd4 WatchSource:0}: Error finding container 28a533c26f06088c258ed8d21aef048d240e566a6e224ebfe9e336d879905fd4: Status 404 returned error can't find the container with id 28a533c26f06088c258ed8d21aef048d240e566a6e224ebfe9e336d879905fd4 Sep 08 18:34:26.981045 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.980900 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j9kxx" Sep 08 18:34:26.981045 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:26.980987 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1fecb2_5eea_4a17_9da0_5a8f9fbf8e82.slice/crio-f37ff71ed04595968b3f0332a23288ff26184131af797e2d193c5e6087102fb6 WatchSource:0}: Error finding container f37ff71ed04595968b3f0332a23288ff26184131af797e2d193c5e6087102fb6: Status 404 returned error can't find the container with id f37ff71ed04595968b3f0332a23288ff26184131af797e2d193c5e6087102fb6 Sep 08 18:34:26.989620 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.989545 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" Sep 08 18:34:26.992296 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:26.992274 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa9c205_ae85_4940_ba9a_b9ba08937086.slice/crio-697219c340b7094d8a14858aab77b7b0e8c0c5f4946528f1ee815b611ba26aa3 WatchSource:0}: Error finding container 697219c340b7094d8a14858aab77b7b0e8c0c5f4946528f1ee815b611ba26aa3: Status 404 returned error can't find the container with id 697219c340b7094d8a14858aab77b7b0e8c0c5f4946528f1ee815b611ba26aa3 Sep 08 18:34:26.998615 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:26.998597 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:26.999651 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:26.999590 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffd0529_39cd_43ca_96d5_6fde24d932db.slice/crio-5a4a8048c8a3fb10dc7cf7d0aa1404ba02fc105b30cf9ecf357a5a27190daf2b WatchSource:0}: Error finding container 5a4a8048c8a3fb10dc7cf7d0aa1404ba02fc105b30cf9ecf357a5a27190daf2b: Status 404 returned error can't find the container with id 5a4a8048c8a3fb10dc7cf7d0aa1404ba02fc105b30cf9ecf357a5a27190daf2b Sep 08 18:34:27.004238 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:27.004219 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jq7qc" Sep 08 18:34:27.230816 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:27.230739 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqp9\" (UniqueName: \"kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9\") pod \"network-check-target-bwbtk\" (UID: \"b5899806-e48c-4bdf-a215-12a2e468f694\") " pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:27.230967 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:27.230867 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 18:34:27.230967 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:27.230885 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 18:34:27.230967 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:27.230898 2553 projected.go:196] Error preparing data for projected volume kube-api-access-bgqp9 for pod openshift-network-diagnostics/network-check-target-bwbtk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:27.230967 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:27.230952 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9 podName:b5899806-e48c-4bdf-a215-12a2e468f694 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:28.230935945 +0000 UTC m=+3.936824177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgqp9" (UniqueName: "kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9") pod "network-check-target-bwbtk" (UID: "b5899806-e48c-4bdf-a215-12a2e468f694") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:27.331684 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:27.331652 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:27.331870 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:27.331715 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:27.331870 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:27.331805 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:27.331870 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:27.331821 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:27.331870 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:27.331860 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs podName:40b6e42c-6d96-445d-8b83-108320730adb nodeName:}" failed. No retries permitted until 2026-09-08 18:34:28.331845111 +0000 UTC m=+4.037733336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs") pod "network-metrics-daemon-sjrbw" (UID: "40b6e42c-6d96-445d-8b83-108320730adb") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:27.332032 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:27.331887 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret podName:f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c nodeName:}" failed. No retries permitted until 2026-09-08 18:34:28.331868111 +0000 UTC m=+4.037756323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret") pod "global-pull-secret-syncer-tzbd8" (UID: "f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c") : object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:27.656716 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:27.656607 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 18:29:27 +0000 UTC" deadline="2028-06-11 17:22:09.970149898 +0000 UTC" Sep 08 18:34:27.656716 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:27.656661 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15406h47m42.313493833s" Sep 08 18:34:27.738715 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:27.738685 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:27.738875 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:27.738804 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:27.753682 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:27.753645 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" event={"ID":"effd0529-39cd-43ca-96d5-6fde24d932db","Type":"ContainerStarted","Data":"5a4a8048c8a3fb10dc7cf7d0aa1404ba02fc105b30cf9ecf357a5a27190daf2b"} Sep 08 18:34:27.754982 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:27.754945 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j9kxx" event={"ID":"7aa9c205-ae85-4940-ba9a-b9ba08937086","Type":"ContainerStarted","Data":"697219c340b7094d8a14858aab77b7b0e8c0c5f4946528f1ee815b611ba26aa3"} Sep 08 18:34:27.756288 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:27.756254 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gchnh" event={"ID":"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7","Type":"ContainerStarted","Data":"28a533c26f06088c258ed8d21aef048d240e566a6e224ebfe9e336d879905fd4"} Sep 08 18:34:27.757525 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:27.757479 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7pnc2" event={"ID":"aa2faa21-f7f1-4010-851b-bd9244d70ea5","Type":"ContainerStarted","Data":"71dee676833decdd0c1ad6dae9c50ab78205c57105150c8f5b6278364731251d"} Sep 08 18:34:27.758621 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:27.758596 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xmpj8" event={"ID":"96b33d5d-d099-464a-adcc-03562ae049c0","Type":"ContainerStarted","Data":"1d06911d1448e87bbe7bb1c23fabacb475a8fda26301b2b1958b8716a3da41be"} Sep 08 18:34:27.759891 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:27.759861 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xpb2v" event={"ID":"df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82","Type":"ContainerStarted","Data":"f37ff71ed04595968b3f0332a23288ff26184131af797e2d193c5e6087102fb6"} Sep 08 18:34:27.761016 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:27.760989 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-klsxq" event={"ID":"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb","Type":"ContainerStarted","Data":"e5465c12f1d483cd063d5ad520c89c3b30f7980d292b7446edc5bff258c42cd9"} Sep 08 18:34:28.230486 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:28.229821 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d4c356_520e_4dd1_b36d_40197469dd3c.slice/crio-9f52971ce5d839fdd13f9b1e27f4146d6be7d766c79584ee889c5728a144ac7b WatchSource:0}: Error finding container 9f52971ce5d839fdd13f9b1e27f4146d6be7d766c79584ee889c5728a144ac7b: Status 404 returned error can't find the container with id 9f52971ce5d839fdd13f9b1e27f4146d6be7d766c79584ee889c5728a144ac7b Sep 08 18:34:28.232925 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:34:28.232898 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1911e404_7356_4fd2_86af_acc9012c5a4b.slice/crio-42ea1ea5ae25f8f0660ad0a29518369a833bfea5aaa5b721cc3f0501a1536bb0 WatchSource:0}: Error finding container 42ea1ea5ae25f8f0660ad0a29518369a833bfea5aaa5b721cc3f0501a1536bb0: Status 404 returned error can't find the container with id 42ea1ea5ae25f8f0660ad0a29518369a833bfea5aaa5b721cc3f0501a1536bb0 Sep 08 18:34:28.240396 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:28.240345 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqp9\" (UniqueName: \"kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9\") pod \"network-check-target-bwbtk\" (UID: \"b5899806-e48c-4bdf-a215-12a2e468f694\") " pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:28.240604 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:28.240585 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 18:34:28.240672 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:28.240617 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 18:34:28.240672 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:28.240631 2553 projected.go:196] Error preparing data for projected volume kube-api-access-bgqp9 for pod openshift-network-diagnostics/network-check-target-bwbtk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:28.240785 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:28.240691 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9 podName:b5899806-e48c-4bdf-a215-12a2e468f694 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:30.240669995 +0000 UTC m=+5.946558222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgqp9" (UniqueName: "kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9") pod "network-check-target-bwbtk" (UID: "b5899806-e48c-4bdf-a215-12a2e468f694") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:28.341053 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:28.341019 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:28.341183 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:28.341129 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:28.341249 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:28.341205 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:28.341304 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:28.341276 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret podName:f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c nodeName:}" failed. No retries permitted until 2026-09-08 18:34:30.3412552 +0000 UTC m=+6.047143417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret") pod "global-pull-secret-syncer-tzbd8" (UID: "f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c") : object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:28.341304 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:28.341211 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:28.341418 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:28.341339 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs podName:40b6e42c-6d96-445d-8b83-108320730adb nodeName:}" failed. No retries permitted until 2026-09-08 18:34:30.341326105 +0000 UTC m=+6.047214333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs") pod "network-metrics-daemon-sjrbw" (UID: "40b6e42c-6d96-445d-8b83-108320730adb") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:28.657539 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:28.657442 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 18:29:27 +0000 UTC" deadline="2028-05-23 10:03:03.190813782 +0000 UTC" Sep 08 18:34:28.657539 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:28.657477 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14943h28m34.533340535s" Sep 08 18:34:28.740079 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:28.740003 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:28.740267 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:28.740143 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:28.740666 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:28.740453 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:28.740666 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:28.740571 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:28.772810 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:28.770921 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" event={"ID":"20d4c356-520e-4dd1-b36d-40197469dd3c","Type":"ContainerStarted","Data":"9f52971ce5d839fdd13f9b1e27f4146d6be7d766c79584ee889c5728a144ac7b"} Sep 08 18:34:28.783739 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:28.783709 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" event={"ID":"1c0a1ff402e4c440e263b451c0fbfb11","Type":"ContainerStarted","Data":"13a1aaeac419ce6a8a73c1873210d875f842e20c79569b524eea9b369e0cd57f"} Sep 08 18:34:28.794223 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:28.794198 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jq7qc" event={"ID":"1911e404-7356-4fd2-86af-acc9012c5a4b","Type":"ContainerStarted","Data":"42ea1ea5ae25f8f0660ad0a29518369a833bfea5aaa5b721cc3f0501a1536bb0"} Sep 08 18:34:29.739242 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:29.739210 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:29.739863 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:29.739356 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:29.814237 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:29.814198 2553 generic.go:356] "Generic (PLEG): container finished" podID="8737f89cceb63fd2f10ea539aa22dbf1" containerID="2e7afa3820fdcd220737d7b43a780d5ffa643943633be25e71b5ab1f9a9880f9" exitCode=0 Sep 08 18:34:29.823710 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:29.823522 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" event={"ID":"8737f89cceb63fd2f10ea539aa22dbf1","Type":"ContainerDied","Data":"2e7afa3820fdcd220737d7b43a780d5ffa643943633be25e71b5ab1f9a9880f9"} Sep 08 18:34:29.840570 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:29.840514 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-43.ec2.internal" podStartSLOduration=3.840494283 podStartE2EDuration="3.840494283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:28.800014004 +0000 UTC m=+4.505902240" watchObservedRunningTime="2026-09-08 18:34:29.840494283 +0000 UTC m=+5.546382523" Sep 08 18:34:30.259450 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:30.259398 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqp9\" (UniqueName: \"kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9\") pod \"network-check-target-bwbtk\" (UID: \"b5899806-e48c-4bdf-a215-12a2e468f694\") " pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:30.259627 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:30.259572 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 18:34:30.259627 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:30.259591 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 18:34:30.259627 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:30.259606 2553 projected.go:196] Error preparing data for projected volume kube-api-access-bgqp9 for pod openshift-network-diagnostics/network-check-target-bwbtk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:30.259802 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:30.259667 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9 podName:b5899806-e48c-4bdf-a215-12a2e468f694 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:34.259646185 +0000 UTC m=+9.965534416 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgqp9" (UniqueName: "kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9") pod "network-check-target-bwbtk" (UID: "b5899806-e48c-4bdf-a215-12a2e468f694") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:30.360479 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:30.359960 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:30.360479 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:30.360010 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:30.360479 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:30.360130 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:30.360479 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:30.360187 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret podName:f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c nodeName:}" failed. No retries permitted until 2026-09-08 18:34:34.360168446 +0000 UTC m=+10.066056674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret") pod "global-pull-secret-syncer-tzbd8" (UID: "f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c") : object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:30.360837 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:30.360593 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:30.360837 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:30.360672 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs podName:40b6e42c-6d96-445d-8b83-108320730adb nodeName:}" failed. No retries permitted until 2026-09-08 18:34:34.360631705 +0000 UTC m=+10.066519935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs") pod "network-metrics-daemon-sjrbw" (UID: "40b6e42c-6d96-445d-8b83-108320730adb") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:30.738737 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:30.738631 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:30.738899 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:30.738761 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:30.740326 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:30.739922 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:30.740326 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:30.740065 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:31.738721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:31.738681 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:31.738907 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:31.738821 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:32.738741 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:32.738710 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:32.739182 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:32.738839 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:32.739182 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:32.738999 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:32.739182 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:32.739104 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:33.738823 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:33.738752 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:33.739196 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:33.738885 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:34.292605 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:34.292566 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqp9\" (UniqueName: \"kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9\") pod \"network-check-target-bwbtk\" (UID: \"b5899806-e48c-4bdf-a215-12a2e468f694\") " pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:34.292772 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:34.292749 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 18:34:34.292838 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:34.292776 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 18:34:34.292838 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:34.292786 2553 projected.go:196] Error preparing data for projected volume kube-api-access-bgqp9 for pod openshift-network-diagnostics/network-check-target-bwbtk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:34.292917 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:34.292845 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9 podName:b5899806-e48c-4bdf-a215-12a2e468f694 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:42.292827325 +0000 UTC m=+17.998715562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgqp9" (UniqueName: "kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9") pod "network-check-target-bwbtk" (UID: "b5899806-e48c-4bdf-a215-12a2e468f694") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:34.393289 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:34.393249 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:34.393452 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:34.393308 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:34.393452 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:34.393390 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:34.393452 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:34.393431 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:34.393596 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:34.393470 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs podName:40b6e42c-6d96-445d-8b83-108320730adb nodeName:}" failed. No retries permitted until 2026-09-08 18:34:42.393448883 +0000 UTC m=+18.099337115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs") pod "network-metrics-daemon-sjrbw" (UID: "40b6e42c-6d96-445d-8b83-108320730adb") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:34.393596 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:34.393492 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret podName:f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c nodeName:}" failed. No retries permitted until 2026-09-08 18:34:42.393480741 +0000 UTC m=+18.099368954 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret") pod "global-pull-secret-syncer-tzbd8" (UID: "f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c") : object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:34.739825 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:34.739709 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:34.740258 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:34.739818 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:34.740329 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:34.740281 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:34.740409 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:34.740383 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:35.740063 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:35.739532 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:35.740063 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:35.739676 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:36.739719 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:36.739247 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:36.739719 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:36.739279 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:36.739719 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:36.739357 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:36.739719 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:36.739478 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:37.738708 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:37.738678 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:37.739053 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:37.738782 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:38.738872 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:38.738838 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:38.738872 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:38.738858 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:38.739300 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:38.738966 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:38.739300 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:38.739040 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:39.739070 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:39.739035 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:39.739477 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:39.739159 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:40.738792 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:40.738750 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:40.738951 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:40.738883 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:40.738951 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:40.738936 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:40.739072 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:40.739050 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:41.739004 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:41.738974 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:41.739376 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:41.739080 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:42.352108 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:42.352069 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqp9\" (UniqueName: \"kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9\") pod \"network-check-target-bwbtk\" (UID: \"b5899806-e48c-4bdf-a215-12a2e468f694\") " pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:42.352304 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:42.352264 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 18:34:42.352304 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:42.352292 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 18:34:42.352304 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:42.352306 2553 projected.go:196] Error preparing data for projected volume kube-api-access-bgqp9 for pod openshift-network-diagnostics/network-check-target-bwbtk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:42.352492 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:42.352390 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9 podName:b5899806-e48c-4bdf-a215-12a2e468f694 nodeName:}" failed. No retries permitted until 2026-09-08 18:34:58.352354084 +0000 UTC m=+34.058242312 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgqp9" (UniqueName: "kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9") pod "network-check-target-bwbtk" (UID: "b5899806-e48c-4bdf-a215-12a2e468f694") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:42.453027 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:42.452990 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:42.453027 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:42.453036 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:42.453231 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:42.453160 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:42.453231 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:42.453208 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret podName:f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c nodeName:}" failed. No retries permitted until 2026-09-08 18:34:58.453194936 +0000 UTC m=+34.159083159 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret") pod "global-pull-secret-syncer-tzbd8" (UID: "f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c") : object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:42.453329 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:42.453161 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:42.453329 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:42.453300 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs podName:40b6e42c-6d96-445d-8b83-108320730adb nodeName:}" failed. No retries permitted until 2026-09-08 18:34:58.453279646 +0000 UTC m=+34.159167874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs") pod "network-metrics-daemon-sjrbw" (UID: "40b6e42c-6d96-445d-8b83-108320730adb") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:42.739104 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:42.739024 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:42.739104 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:42.739058 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:42.739579 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:42.739160 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:42.739579 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:42.739297 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:43.738698 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:43.738660 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:43.738895 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:43.738772 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:44.739828 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:44.739789 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:44.740275 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:44.739875 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:44.740275 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:44.739884 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:44.740275 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:44.739991 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:45.739687 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.739535 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:45.739788 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:45.739771 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:45.854620 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.854440 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" event={"ID":"8737f89cceb63fd2f10ea539aa22dbf1","Type":"ContainerStarted","Data":"42839e3e01bd9a8327863844f4555127b662fa31bcfcc71e7c231a397983d97d"} Sep 08 18:34:45.856229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.856202 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" event={"ID":"effd0529-39cd-43ca-96d5-6fde24d932db","Type":"ContainerStarted","Data":"dfe75229282f0db019371ac1e34c1eddb497279e0435637477ce8a87ce274427"} Sep 08 18:34:45.859928 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.859897 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j9kxx" event={"ID":"7aa9c205-ae85-4940-ba9a-b9ba08937086","Type":"ContainerStarted","Data":"57bee06bc99a20dfb14f72509e5e504d87bcc77d56228764c16ab05f37bfb429"} Sep 08 18:34:45.861387 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.861330 2553 generic.go:356] "Generic (PLEG): container finished" podID="d8e54f1e-5ce8-481f-a6cb-c01a98251ed7" containerID="ee6d9fdd56a97ec1fd655f3dda4e0f37e44327837930f4d7036a26b4696f1323" exitCode=0 Sep 08 18:34:45.861469 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.861443 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gchnh" event={"ID":"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7","Type":"ContainerDied","Data":"ee6d9fdd56a97ec1fd655f3dda4e0f37e44327837930f4d7036a26b4696f1323"} Sep 08 18:34:45.863061 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.863039 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7pnc2" event={"ID":"aa2faa21-f7f1-4010-851b-bd9244d70ea5","Type":"ContainerStarted","Data":"cade1f757cae4fe19b01d2293bc30bfe1f9692fb1d07791110f9a2ec8f520acd"} Sep 08 18:34:45.864511 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.864486 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xmpj8" event={"ID":"96b33d5d-d099-464a-adcc-03562ae049c0","Type":"ContainerStarted","Data":"8106eabd430308e83c7e7a2a36a50582b262f73f91c7c9bae614a7498f395907"} Sep 08 18:34:45.866025 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.866001 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jq7qc" event={"ID":"1911e404-7356-4fd2-86af-acc9012c5a4b","Type":"ContainerStarted","Data":"7ba2a3a22e02359098fce7d652cf7d31dd2ad541f8a0fec771be8251832317a8"} Sep 08 18:34:45.867647 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.867619 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-klsxq" event={"ID":"5ba2ecc9-ebf7-4073-849d-c1700b3cecfb","Type":"ContainerStarted","Data":"ec834793646bccc8d8b359ec19a1da415a561234bcd401ddc78d0da2d5ba335f"} Sep 08 18:34:45.868903 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.868867 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-43.ec2.internal" podStartSLOduration=19.868856453 podStartE2EDuration="19.868856453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:45.868511577 +0000 UTC m=+21.574399810" watchObservedRunningTime="2026-09-08 18:34:45.868856453 +0000 UTC m=+21.574744685" Sep 08 18:34:45.870276 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.870259 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 18:34:45.870641 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.870623 2553 generic.go:356] "Generic (PLEG): container finished" podID="20d4c356-520e-4dd1-b36d-40197469dd3c" containerID="970469b4d7748391950a27dedc0f6f971afc2190ae57f3fb6085fd9844cfc930" exitCode=1 Sep 08 18:34:45.870711 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.870651 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" event={"ID":"20d4c356-520e-4dd1-b36d-40197469dd3c","Type":"ContainerStarted","Data":"86ff3d4617f42dc5681b4a03b7e70bf8bd7749ee3c1bf1cbded24468e4be9622"} Sep 08 18:34:45.870711 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.870696 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" event={"ID":"20d4c356-520e-4dd1-b36d-40197469dd3c","Type":"ContainerDied","Data":"970469b4d7748391950a27dedc0f6f971afc2190ae57f3fb6085fd9844cfc930"} Sep 08 18:34:45.870797 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.870726 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" event={"ID":"20d4c356-520e-4dd1-b36d-40197469dd3c","Type":"ContainerStarted","Data":"59e2578499550a5b6ccc952fff62e9670892a7e5ec0a34b24712cc78b2905ad4"} Sep 08 18:34:45.881644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.881606 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xmpj8" podStartSLOduration=2.433948048 podStartE2EDuration="20.881590446s" totalImagesPullingTime="18.447642398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:45.881137156 +0000 UTC m=+21.587025389" watchObservedRunningTime="2026-09-08 18:34:45.881590446 +0000 UTC m=+21.587478679" Sep 08 18:34:45.910272 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.910231 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7pnc2" podStartSLOduration=2.211039725 podStartE2EDuration="20.910214687s" totalImagesPullingTime="18.699174962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:45.896555296 +0000 UTC m=+21.602443529" watchObservedRunningTime="2026-09-08 18:34:45.910214687 +0000 UTC m=+21.616102920" Sep 08 18:34:45.910526 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.910502 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j9kxx" podStartSLOduration=6.40038485 podStartE2EDuration="20.910492434s" totalImagesPullingTime="14.510107584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:45.909900884 +0000 UTC m=+21.615789116" watchObservedRunningTime="2026-09-08 18:34:45.910492434 +0000 UTC m=+21.616380715" Sep 08 18:34:45.935429 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.935387 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-klsxq" podStartSLOduration=2.484452387 podStartE2EDuration="20.9353556s" totalImagesPullingTime="18.450903213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:45.935121949 +0000 UTC m=+21.641010183" watchObservedRunningTime="2026-09-08 18:34:45.9353556 +0000 UTC m=+21.641243831" Sep 08 18:34:45.982483 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:45.982445 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jq7qc" podStartSLOduration=3.839834593 podStartE2EDuration="20.982430402s" totalImagesPullingTime="17.142595809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:45.982357229 +0000 UTC m=+21.688245461" watchObservedRunningTime="2026-09-08 18:34:45.982430402 +0000 UTC m=+21.688318612" Sep 08 18:34:46.739472 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:46.739447 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:46.739621 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:46.739457 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:46.739621 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:46.739549 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:46.739697 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:46.739638 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:46.875012 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:46.874975 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xpb2v" event={"ID":"df1fecb2-5eea-4a17-9da0-5a8f9fbf8e82","Type":"ContainerStarted","Data":"c97538f0cded49b109764b04fe772132b912250bbf45c26f765dd33f6cbf640f"} Sep 08 18:34:46.877714 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:46.877694 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 18:34:46.878125 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:46.878101 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" event={"ID":"20d4c356-520e-4dd1-b36d-40197469dd3c","Type":"ContainerStarted","Data":"5aca00907d6e870e90bed4c53a5d7f73d8f2dd955d5917555552c197d44c9440"} Sep 08 18:34:46.878175 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:46.878165 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" event={"ID":"20d4c356-520e-4dd1-b36d-40197469dd3c","Type":"ContainerStarted","Data":"dcdcf401b8ca326a50e706c0567505157399662cd275662dcf49f51bbe00757c"} Sep 08 18:34:46.878218 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:46.878196 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" event={"ID":"20d4c356-520e-4dd1-b36d-40197469dd3c","Type":"ContainerStarted","Data":"cadb38cbf14339ceed0d41bc47a4b72b32d82c89cad8bff49c862d00c1fa0db2"} Sep 08 18:34:46.892015 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:46.891976 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xpb2v" podStartSLOduration=3.471220414 podStartE2EDuration="21.891963911s" totalImagesPullingTime="18.420743497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:46.891625348 +0000 UTC m=+22.597513580" watchObservedRunningTime="2026-09-08 18:34:46.891963911 +0000 UTC m=+22.597852143" Sep 08 18:34:47.167132 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:47.167110 2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 08 18:34:47.392680 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:47.392600 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j9kxx" Sep 08 18:34:47.393147 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:47.393125 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j9kxx" Sep 08 18:34:47.666391 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:47.666211 2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T18:34:47.167127075Z","UUID":"f8710b19-2b5b-41cc-8300-0789971f4ef1","Handler":null,"Name":"","Endpoint":""} Sep 08 18:34:47.668068 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:47.667923 2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 08 18:34:47.668068 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:47.668074 2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 08 18:34:47.738795 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:47.738764 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:47.738977 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:47.738910 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:47.883666 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:47.883622 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" event={"ID":"effd0529-39cd-43ca-96d5-6fde24d932db","Type":"ContainerStarted","Data":"6d1bc519618a21bfc947956573d423f56aef8fff60ffcb13e9e43f068fe9b102"} Sep 08 18:34:47.884091 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:47.883995 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j9kxx" Sep 08 18:34:47.884554 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:47.884535 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j9kxx" Sep 08 18:34:48.739432 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:48.739333 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:48.739591 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:48.739334 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:48.739591 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:48.739477 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:48.739591 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:48.739535 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:48.889045 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:48.889014 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" event={"ID":"effd0529-39cd-43ca-96d5-6fde24d932db","Type":"ContainerStarted","Data":"90b4e23da0e7076fe0177bc2664d533a3b6ffef09f719f21d5b794f242d9221b"} Sep 08 18:34:48.892877 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:48.892852 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 18:34:48.893278 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:48.893246 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" event={"ID":"20d4c356-520e-4dd1-b36d-40197469dd3c","Type":"ContainerStarted","Data":"c5f76586d984691487cffd3e9b4567f10c24bb71662d821e504ecf1249472f2e"} Sep 08 18:34:48.906448 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:48.906408 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9kqv" podStartSLOduration=2.888555086 podStartE2EDuration="23.906392675s" totalImagesPullingTime="21.017837589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:48.905859509 +0000 UTC m=+24.611747742" watchObservedRunningTime="2026-09-08 18:34:48.906392675 +0000 UTC m=+24.612280909" Sep 08 18:34:49.739117 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:49.739085 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:49.739305 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:49.739212 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:50.739568 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:50.739534 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:50.739972 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:50.739650 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:50.739972 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:50.739698 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:50.739972 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:50.739780 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:51.739223 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:51.739190 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:51.739418 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:51.739320 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:52.739570 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:52.739402 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:52.740116 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:52.739452 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:52.740116 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:52.739635 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:52.740116 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:52.739695 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:52.905801 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:52.905773 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 18:34:52.906153 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:52.906129 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" event={"ID":"20d4c356-520e-4dd1-b36d-40197469dd3c","Type":"ContainerStarted","Data":"434b2bc8834b5d4413b499c21f1f4eb6ba97533c6ebb4b3886979686cbabbd37"} Sep 08 18:34:52.906444 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:52.906346 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:52.906605 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:52.906588 2553 scope.go:117] "RemoveContainer" containerID="970469b4d7748391950a27dedc0f6f971afc2190ae57f3fb6085fd9844cfc930" Sep 08 18:34:52.908178 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:52.908157 2553 generic.go:356] "Generic (PLEG): container finished" podID="d8e54f1e-5ce8-481f-a6cb-c01a98251ed7" containerID="f08a9f0157e689bd78c39d74ed12726987cc964e42194bd22056c2334607ef4f" exitCode=0 Sep 08 18:34:52.908266 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:52.908198 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gchnh" event={"ID":"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7","Type":"ContainerDied","Data":"f08a9f0157e689bd78c39d74ed12726987cc964e42194bd22056c2334607ef4f"} Sep 08 18:34:52.923771 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:52.923552 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:53.738753 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:53.738726 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:53.738905 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:53.738864 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:53.916341 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:53.916312 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 18:34:53.916995 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:53.916746 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" event={"ID":"20d4c356-520e-4dd1-b36d-40197469dd3c","Type":"ContainerStarted","Data":"fca8bf40bd7795aeec72a43b2e071eb3d992494b9fe9c660409c92f8758cf9c3"} Sep 08 18:34:53.917162 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:53.917143 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:53.917249 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:53.917213 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:53.922805 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:53.922602 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tzbd8"] Sep 08 18:34:53.922916 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:53.922899 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:53.923010 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:53.922994 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:53.925618 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:53.925593 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bwbtk"] Sep 08 18:34:53.925721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:53.925682 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:53.925795 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:53.925773 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:53.933896 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:53.933851 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:34:53.941547 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:53.941527 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sjrbw"] Sep 08 18:34:53.941645 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:53.941600 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:53.941701 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:53.941683 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:53.950383 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:53.950327 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" podStartSLOduration=11.734777277 podStartE2EDuration="28.950311922s" totalImagesPullingTime="17.215534645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:34:53.950032461 +0000 UTC m=+29.655920708" watchObservedRunningTime="2026-09-08 18:34:53.950311922 +0000 UTC m=+29.656200155" Sep 08 18:34:54.920849 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:54.920817 2553 generic.go:356] "Generic (PLEG): container finished" podID="d8e54f1e-5ce8-481f-a6cb-c01a98251ed7" containerID="f170857d0f216aff9fdd9481212e0f825aa17af9bfdae33e47a6c083ad1d0328" exitCode=0 Sep 08 18:34:54.921170 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:54.920898 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gchnh" event={"ID":"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7","Type":"ContainerDied","Data":"f170857d0f216aff9fdd9481212e0f825aa17af9bfdae33e47a6c083ad1d0328"} Sep 08 18:34:55.739286 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:55.739249 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:55.739480 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:55.739345 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:55.739480 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:55.739351 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:55.739480 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:55.739394 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:55.739647 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:55.739485 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:55.739647 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:55.739586 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:56.926996 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:56.926950 2553 generic.go:356] "Generic (PLEG): container finished" podID="d8e54f1e-5ce8-481f-a6cb-c01a98251ed7" containerID="b3614b129fd93faea61f05c32ad05b4487755d50832d80980a53d4ffb72900f8" exitCode=0 Sep 08 18:34:56.927471 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:56.927008 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gchnh" event={"ID":"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7","Type":"ContainerDied","Data":"b3614b129fd93faea61f05c32ad05b4487755d50832d80980a53d4ffb72900f8"} Sep 08 18:34:57.738908 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:57.738870 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:57.739095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:57.738880 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:57.739095 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:57.738982 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bwbtk" podUID="b5899806-e48c-4bdf-a215-12a2e468f694" Sep 08 18:34:57.739221 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:57.738881 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:57.739221 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:57.739103 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:34:57.739221 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:57.739196 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzbd8" podUID="f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c" Sep 08 18:34:58.372321 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:58.372276 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqp9\" (UniqueName: \"kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9\") pod \"network-check-target-bwbtk\" (UID: \"b5899806-e48c-4bdf-a215-12a2e468f694\") " pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:58.372771 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:58.372481 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 18:34:58.372771 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:58.372509 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 18:34:58.372771 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:58.372522 2553 projected.go:196] Error preparing data for projected volume kube-api-access-bgqp9 for pod openshift-network-diagnostics/network-check-target-bwbtk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:58.372771 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:58.372597 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9 podName:b5899806-e48c-4bdf-a215-12a2e468f694 nodeName:}" failed. No retries permitted until 2026-09-08 18:35:30.372571594 +0000 UTC m=+66.078459808 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgqp9" (UniqueName: "kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9") pod "network-check-target-bwbtk" (UID: "b5899806-e48c-4bdf-a215-12a2e468f694") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 18:34:58.473428 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:58.473400 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:58.473612 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:58.473438 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:58.473612 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:58.473547 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:58.473612 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:58.473606 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs podName:40b6e42c-6d96-445d-8b83-108320730adb nodeName:}" failed. No retries permitted until 2026-09-08 18:35:30.473591729 +0000 UTC m=+66.179479943 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs") pod "network-metrics-daemon-sjrbw" (UID: "40b6e42c-6d96-445d-8b83-108320730adb") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 18:34:58.473612 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:58.473605 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:58.473762 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:58.473632 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret podName:f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c nodeName:}" failed. No retries permitted until 2026-09-08 18:35:30.473626342 +0000 UTC m=+66.179514551 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret") pod "global-pull-secret-syncer-tzbd8" (UID: "f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c") : object "kube-system"/"original-pull-secret" not registered Sep 08 18:34:59.562182 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.562154 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-43.ec2.internal" event="NodeReady" Sep 08 18:34:59.562657 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.562278 2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 08 18:34:59.604192 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.604160 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pjc2h"] Sep 08 18:34:59.619017 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.618969 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mnblg"] Sep 08 18:34:59.619853 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.619832 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pjc2h" Sep 08 18:34:59.622341 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.622313 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 08 18:34:59.622575 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.622559 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rsh8d\"" Sep 08 18:34:59.623652 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.623601 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 08 18:34:59.634856 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.634839 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pjc2h"] Sep 08 18:34:59.634968 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.634881 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mnblg"] Sep 08 18:34:59.634968 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.634955 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mnblg" Sep 08 18:34:59.638470 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.638065 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9b5jk\"" Sep 08 18:34:59.638470 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.638285 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 08 18:34:59.638610 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.638063 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 08 18:34:59.638950 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.638924 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 08 18:34:59.738874 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.738844 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:34:59.739036 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.738959 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:34:59.739036 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.738994 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:34:59.741853 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.741829 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gmn47\"" Sep 08 18:34:59.741853 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.741845 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 18:34:59.742189 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.742020 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 18:34:59.742189 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.742104 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z79bg\"" Sep 08 18:34:59.742189 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.742112 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 18:34:59.742189 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.742146 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 18:34:59.782343 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.782325 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-tmp-dir\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:34:59.782443 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.782350 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp9qg\" (UniqueName: \"kubernetes.io/projected/2ae8b37f-62ce-4a27-ae4b-369e38c6de40-kube-api-access-mp9qg\") pod \"ingress-canary-mnblg\" (UID: \"2ae8b37f-62ce-4a27-ae4b-369e38c6de40\") " pod="openshift-ingress-canary/ingress-canary-mnblg" Sep 08 18:34:59.782443 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.782402 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-config-volume\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:34:59.782514 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.782473 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvppg\" (UniqueName: \"kubernetes.io/projected/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-kube-api-access-vvppg\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:34:59.782514 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.782498 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:34:59.782581 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.782520 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ae8b37f-62ce-4a27-ae4b-369e38c6de40-cert\") pod \"ingress-canary-mnblg\" (UID: \"2ae8b37f-62ce-4a27-ae4b-369e38c6de40\") " pod="openshift-ingress-canary/ingress-canary-mnblg" Sep 08 18:34:59.883559 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.883531 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-config-volume\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:34:59.883708 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.883581 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvppg\" (UniqueName: \"kubernetes.io/projected/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-kube-api-access-vvppg\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:34:59.883708 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.883609 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:34:59.883708 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.883643 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ae8b37f-62ce-4a27-ae4b-369e38c6de40-cert\") pod \"ingress-canary-mnblg\" (UID: \"2ae8b37f-62ce-4a27-ae4b-369e38c6de40\") " pod="openshift-ingress-canary/ingress-canary-mnblg" Sep 08 18:34:59.883708 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.883683 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-tmp-dir\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:34:59.883708 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.883705 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp9qg\" (UniqueName: \"kubernetes.io/projected/2ae8b37f-62ce-4a27-ae4b-369e38c6de40-kube-api-access-mp9qg\") pod \"ingress-canary-mnblg\" (UID: \"2ae8b37f-62ce-4a27-ae4b-369e38c6de40\") " pod="openshift-ingress-canary/ingress-canary-mnblg" Sep 08 18:34:59.883954 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:59.883928 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:34:59.884009 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:34:59.884002 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls podName:71e540dd-1e1b-46ab-b6d5-01a0ccf6463c nodeName:}" failed. No retries permitted until 2026-09-08 18:35:00.383983871 +0000 UTC m=+36.089872104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls") pod "dns-default-pjc2h" (UID: "71e540dd-1e1b-46ab-b6d5-01a0ccf6463c") : secret "dns-default-metrics-tls" not found Sep 08 18:34:59.884101 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.884074 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-config-volume\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:34:59.884334 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.884309 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-tmp-dir\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:34:59.888445 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.888419 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ae8b37f-62ce-4a27-ae4b-369e38c6de40-cert\") pod \"ingress-canary-mnblg\" (UID: \"2ae8b37f-62ce-4a27-ae4b-369e38c6de40\") " pod="openshift-ingress-canary/ingress-canary-mnblg" Sep 08 18:34:59.893036 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.893011 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvppg\" (UniqueName: \"kubernetes.io/projected/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-kube-api-access-vvppg\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:34:59.893329 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.893309 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp9qg\" (UniqueName: \"kubernetes.io/projected/2ae8b37f-62ce-4a27-ae4b-369e38c6de40-kube-api-access-mp9qg\") pod \"ingress-canary-mnblg\" (UID: \"2ae8b37f-62ce-4a27-ae4b-369e38c6de40\") " pod="openshift-ingress-canary/ingress-canary-mnblg" Sep 08 18:34:59.945776 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:34:59.945754 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mnblg" Sep 08 18:35:00.264754 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:00.264015 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mnblg"] Sep 08 18:35:00.386229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:00.386197 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:35:00.386420 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:35:00.386304 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:35:00.386420 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:35:00.386383 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls podName:71e540dd-1e1b-46ab-b6d5-01a0ccf6463c nodeName:}" failed. No retries permitted until 2026-09-08 18:35:01.386354965 +0000 UTC m=+37.092243180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls") pod "dns-default-pjc2h" (UID: "71e540dd-1e1b-46ab-b6d5-01a0ccf6463c") : secret "dns-default-metrics-tls" not found Sep 08 18:35:01.393295 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:01.393260 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:35:01.393816 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:35:01.393448 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:35:01.393816 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:35:01.393522 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls podName:71e540dd-1e1b-46ab-b6d5-01a0ccf6463c nodeName:}" failed. No retries permitted until 2026-09-08 18:35:03.393501163 +0000 UTC m=+39.099389375 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls") pod "dns-default-pjc2h" (UID: "71e540dd-1e1b-46ab-b6d5-01a0ccf6463c") : secret "dns-default-metrics-tls" not found Sep 08 18:35:01.679187 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:35:01.679125 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae8b37f_62ce_4a27_ae4b_369e38c6de40.slice/crio-96f8644226aaa6198ea9e2ff4287c640e519f4f47e3b0ece55dda843553a9dbb WatchSource:0}: Error finding container 96f8644226aaa6198ea9e2ff4287c640e519f4f47e3b0ece55dda843553a9dbb: Status 404 returned error can't find the container with id 96f8644226aaa6198ea9e2ff4287c640e519f4f47e3b0ece55dda843553a9dbb Sep 08 18:35:01.940827 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:01.940801 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mnblg" event={"ID":"2ae8b37f-62ce-4a27-ae4b-369e38c6de40","Type":"ContainerStarted","Data":"96f8644226aaa6198ea9e2ff4287c640e519f4f47e3b0ece55dda843553a9dbb"} Sep 08 18:35:02.946601 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:02.946344 2553 generic.go:356] "Generic (PLEG): container finished" podID="d8e54f1e-5ce8-481f-a6cb-c01a98251ed7" containerID="64ab5c856cf427c7957009790982939c0419bcd58ca0c2f30a0ac3bd23daaa38" exitCode=0 Sep 08 18:35:02.946601 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:02.946401 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gchnh" event={"ID":"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7","Type":"ContainerDied","Data":"64ab5c856cf427c7957009790982939c0419bcd58ca0c2f30a0ac3bd23daaa38"} Sep 08 18:35:03.409636 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:03.409593 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:35:03.409843 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:35:03.409764 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:35:03.409916 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:35:03.409866 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls podName:71e540dd-1e1b-46ab-b6d5-01a0ccf6463c nodeName:}" failed. No retries permitted until 2026-09-08 18:35:07.40984134 +0000 UTC m=+43.115729568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls") pod "dns-default-pjc2h" (UID: "71e540dd-1e1b-46ab-b6d5-01a0ccf6463c") : secret "dns-default-metrics-tls" not found Sep 08 18:35:03.953437 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:03.953400 2553 generic.go:356] "Generic (PLEG): container finished" podID="d8e54f1e-5ce8-481f-a6cb-c01a98251ed7" containerID="38616a215ab591403f664bedabba4615c816d79c0da0078bb0c6767f205dcec8" exitCode=0 Sep 08 18:35:03.953944 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:03.953443 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gchnh" event={"ID":"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7","Type":"ContainerDied","Data":"38616a215ab591403f664bedabba4615c816d79c0da0078bb0c6767f205dcec8"} Sep 08 18:35:04.959435 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:04.959398 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gchnh" event={"ID":"d8e54f1e-5ce8-481f-a6cb-c01a98251ed7","Type":"ContainerStarted","Data":"120cdb546d876e0a904afd45ce1b08514e07db957a6eae2c3c0cc73742d0d7f2"} Sep 08 18:35:04.960788 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:04.960762 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mnblg" event={"ID":"2ae8b37f-62ce-4a27-ae4b-369e38c6de40","Type":"ContainerStarted","Data":"390d71f4bbb37c4f0f9189057139b86f01a45425c021a7ce3d4fe85690cd4c53"} Sep 08 18:35:04.980003 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:04.979955 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gchnh" podStartSLOduration=8.266936627 podStartE2EDuration="39.979941923s" totalImagesPullingTime="31.713005296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:35:04.979753927 +0000 UTC m=+40.685642158" watchObservedRunningTime="2026-09-08 18:35:04.979941923 +0000 UTC m=+40.685830155" Sep 08 18:35:04.997686 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:04.997624 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mnblg" podStartSLOduration=3.150498861 podStartE2EDuration="5.997604905s" totalImagesPullingTime="2.847106044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:35:04.996676159 +0000 UTC m=+40.702564393" watchObservedRunningTime="2026-09-08 18:35:04.997604905 +0000 UTC m=+40.703493138" Sep 08 18:35:07.437719 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:07.437683 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:35:07.438089 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:35:07.437825 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:35:07.438089 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:35:07.437901 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls podName:71e540dd-1e1b-46ab-b6d5-01a0ccf6463c nodeName:}" failed. No retries permitted until 2026-09-08 18:35:15.43788373 +0000 UTC m=+51.143771960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls") pod "dns-default-pjc2h" (UID: "71e540dd-1e1b-46ab-b6d5-01a0ccf6463c") : secret "dns-default-metrics-tls" not found Sep 08 18:35:15.493875 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:15.493835 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:35:15.494310 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:35:15.493959 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:35:15.494310 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:35:15.494039 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls podName:71e540dd-1e1b-46ab-b6d5-01a0ccf6463c nodeName:}" failed. No retries permitted until 2026-09-08 18:35:31.494022253 +0000 UTC m=+67.199910478 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls") pod "dns-default-pjc2h" (UID: "71e540dd-1e1b-46ab-b6d5-01a0ccf6463c") : secret "dns-default-metrics-tls" not found Sep 08 18:35:25.935972 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:25.935943 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f4hs5" Sep 08 18:35:30.396163 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:30.396110 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqp9\" (UniqueName: \"kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9\") pod \"network-check-target-bwbtk\" (UID: \"b5899806-e48c-4bdf-a215-12a2e468f694\") " pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:35:30.399424 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:30.399406 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 18:35:30.409886 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:30.409870 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 18:35:30.420851 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:30.420830 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqp9\" (UniqueName: \"kubernetes.io/projected/b5899806-e48c-4bdf-a215-12a2e468f694-kube-api-access-bgqp9\") pod \"network-check-target-bwbtk\" (UID: \"b5899806-e48c-4bdf-a215-12a2e468f694\") " pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:35:30.497059 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:30.497017 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:35:30.497161 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:30.497082 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:35:30.499929 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:30.499909 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 18:35:30.500017 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:30.499946 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 18:35:30.508160 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:35:30.508139 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 08 18:35:30.508247 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:35:30.508210 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs podName:40b6e42c-6d96-445d-8b83-108320730adb nodeName:}" failed. No retries permitted until 2026-09-08 18:36:34.508192285 +0000 UTC m=+130.214080494 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs") pod "network-metrics-daemon-sjrbw" (UID: "40b6e42c-6d96-445d-8b83-108320730adb") : secret "metrics-daemon-secret" not found Sep 08 18:35:30.510263 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:30.510243 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c-original-pull-secret\") pod \"global-pull-secret-syncer-tzbd8\" (UID: \"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c\") " pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:35:30.662755 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:30.662673 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gmn47\"" Sep 08 18:35:30.665732 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:30.665715 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzbd8" Sep 08 18:35:30.670664 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:30.670643 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:35:30.941329 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:30.941253 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tzbd8"] Sep 08 18:35:30.944247 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:30.944219 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bwbtk"] Sep 08 18:35:30.944247 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:35:30.944222 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4bc5cda_5c62_4dae_8e79_d6f3f81bf03c.slice/crio-4323b980ea54e0646683c5a59e472a85b5d0ed3e65dda775693097f278a4b613 WatchSource:0}: Error finding container 4323b980ea54e0646683c5a59e472a85b5d0ed3e65dda775693097f278a4b613: Status 404 returned error can't find the container with id 4323b980ea54e0646683c5a59e472a85b5d0ed3e65dda775693097f278a4b613 Sep 08 18:35:30.953771 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:35:30.953750 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5899806_e48c_4bdf_a215_12a2e468f694.slice/crio-59ef681f254d15ea6da80e28fe1ceedf0958a508f2635fd7459eec35de915968 WatchSource:0}: Error finding container 59ef681f254d15ea6da80e28fe1ceedf0958a508f2635fd7459eec35de915968: Status 404 returned error can't find the container with id 59ef681f254d15ea6da80e28fe1ceedf0958a508f2635fd7459eec35de915968 Sep 08 18:35:31.023077 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:31.023042 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bwbtk" event={"ID":"b5899806-e48c-4bdf-a215-12a2e468f694","Type":"ContainerStarted","Data":"59ef681f254d15ea6da80e28fe1ceedf0958a508f2635fd7459eec35de915968"} Sep 08 18:35:31.024059 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:31.024037 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tzbd8" event={"ID":"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c","Type":"ContainerStarted","Data":"4323b980ea54e0646683c5a59e472a85b5d0ed3e65dda775693097f278a4b613"} Sep 08 18:35:31.503734 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:31.503687 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:35:31.504202 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:35:31.503840 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:35:31.504202 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:35:31.503911 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls podName:71e540dd-1e1b-46ab-b6d5-01a0ccf6463c nodeName:}" failed. No retries permitted until 2026-09-08 18:36:03.50389298 +0000 UTC m=+99.209781210 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls") pod "dns-default-pjc2h" (UID: "71e540dd-1e1b-46ab-b6d5-01a0ccf6463c") : secret "dns-default-metrics-tls" not found Sep 08 18:35:36.040065 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:36.040025 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bwbtk" event={"ID":"b5899806-e48c-4bdf-a215-12a2e468f694","Type":"ContainerStarted","Data":"4b0e87fe35829d4d9cbaf7639155d001f2091f9a7f94992ce5763c07d15639c0"} Sep 08 18:35:36.040697 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:36.040140 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:35:36.041701 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:36.041675 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tzbd8" event={"ID":"f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c","Type":"ContainerStarted","Data":"aee28c86e33e0ea6fdc2d368bd6e42859a32d7fa516496132737707a56cfe731"} Sep 08 18:35:36.054634 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:36.054590 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bwbtk" podStartSLOduration=66.194921601 podStartE2EDuration="1m11.05457842s" totalImagesPullingTime="4.859656819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:35:36.05361055 +0000 UTC m=+71.759498783" watchObservedRunningTime="2026-09-08 18:35:36.05457842 +0000 UTC m=+71.760466647" Sep 08 18:35:36.067219 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:36.067139 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tzbd8" podStartSLOduration=66.195340982 podStartE2EDuration="1m11.067127085s" totalImagesPullingTime="4.871786103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:35:36.06651425 +0000 UTC m=+71.772402473" watchObservedRunningTime="2026-09-08 18:35:36.067127085 +0000 UTC m=+71.773015318" Sep 08 18:35:55.977337 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:55.977303 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq"] Sep 08 18:35:55.979343 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:55.979326 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq" Sep 08 18:35:55.983187 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:55.983163 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 08 18:35:55.983299 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:55.983166 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 08 18:35:55.983445 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:55.983428 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 08 18:35:55.983504 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:55.983444 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-np2fl\"" Sep 08 18:35:55.983560 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:55.983514 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 08 18:35:55.989897 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:55.989877 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq"] Sep 08 18:35:56.070485 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:56.070447 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2j8s\" (UniqueName: \"kubernetes.io/projected/0a006bbd-7ef8-4605-860f-762a7c707723-kube-api-access-w2j8s\") pod \"managed-serviceaccount-addon-agent-6458cc6948-hktdq\" (UID: \"0a006bbd-7ef8-4605-860f-762a7c707723\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq" Sep 08 18:35:56.070617 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:56.070502 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a006bbd-7ef8-4605-860f-762a7c707723-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6458cc6948-hktdq\" (UID: \"0a006bbd-7ef8-4605-860f-762a7c707723\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq" Sep 08 18:35:56.171416 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:56.171353 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2j8s\" (UniqueName: \"kubernetes.io/projected/0a006bbd-7ef8-4605-860f-762a7c707723-kube-api-access-w2j8s\") pod \"managed-serviceaccount-addon-agent-6458cc6948-hktdq\" (UID: \"0a006bbd-7ef8-4605-860f-762a7c707723\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq" Sep 08 18:35:56.171574 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:56.171522 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a006bbd-7ef8-4605-860f-762a7c707723-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6458cc6948-hktdq\" (UID: \"0a006bbd-7ef8-4605-860f-762a7c707723\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq" Sep 08 18:35:56.173953 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:56.173930 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a006bbd-7ef8-4605-860f-762a7c707723-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6458cc6948-hktdq\" (UID: \"0a006bbd-7ef8-4605-860f-762a7c707723\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq" Sep 08 18:35:56.178761 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:56.178736 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2j8s\" (UniqueName: \"kubernetes.io/projected/0a006bbd-7ef8-4605-860f-762a7c707723-kube-api-access-w2j8s\") pod \"managed-serviceaccount-addon-agent-6458cc6948-hktdq\" (UID: \"0a006bbd-7ef8-4605-860f-762a7c707723\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq" Sep 08 18:35:56.301173 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:56.301085 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq" Sep 08 18:35:56.565290 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:56.565213 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq"] Sep 08 18:35:56.568517 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:35:56.568486 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a006bbd_7ef8_4605_860f_762a7c707723.slice/crio-9648ec23b20d48874929ca2e469d5cdc1ad65b5c44e6fd91cf3acd69623dbe45 WatchSource:0}: Error finding container 9648ec23b20d48874929ca2e469d5cdc1ad65b5c44e6fd91cf3acd69623dbe45: Status 404 returned error can't find the container with id 9648ec23b20d48874929ca2e469d5cdc1ad65b5c44e6fd91cf3acd69623dbe45 Sep 08 18:35:57.092724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:35:57.092690 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq" event={"ID":"0a006bbd-7ef8-4605-860f-762a7c707723","Type":"ContainerStarted","Data":"9648ec23b20d48874929ca2e469d5cdc1ad65b5c44e6fd91cf3acd69623dbe45"} Sep 08 18:36:00.101487 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:36:00.101452 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq" event={"ID":"0a006bbd-7ef8-4605-860f-762a7c707723","Type":"ContainerStarted","Data":"73b7b45c4b4f0b2f0dc6c6e483df3e94daca9baa824fb622798b0b416bbc2d6f"} Sep 08 18:36:00.115753 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:36:00.115712 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq" podStartSLOduration=2.351302615 podStartE2EDuration="5.115698089s" totalImagesPullingTime="2.764395474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:35:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:36:00.11553347 +0000 UTC m=+95.821421702" watchObservedRunningTime="2026-09-08 18:36:00.115698089 +0000 UTC m=+95.821586321" Sep 08 18:36:03.515172 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:36:03.515133 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:36:03.515479 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:36:03.515287 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:36:03.515479 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:36:03.515353 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls podName:71e540dd-1e1b-46ab-b6d5-01a0ccf6463c nodeName:}" failed. No retries permitted until 2026-09-08 18:37:07.515337908 +0000 UTC m=+163.221226132 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls") pod "dns-default-pjc2h" (UID: "71e540dd-1e1b-46ab-b6d5-01a0ccf6463c") : secret "dns-default-metrics-tls" not found Sep 08 18:36:07.047128 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:36:07.047094 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bwbtk" Sep 08 18:36:34.519573 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:36:34.519520 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:36:34.520060 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:36:34.519683 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 08 18:36:34.520060 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:36:34.519756 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs podName:40b6e42c-6d96-445d-8b83-108320730adb nodeName:}" failed. No retries permitted until 2026-09-08 18:38:36.519739587 +0000 UTC m=+252.225627800 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs") pod "network-metrics-daemon-sjrbw" (UID: "40b6e42c-6d96-445d-8b83-108320730adb") : secret "metrics-daemon-secret" not found Sep 08 18:37:02.631544 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:37:02.631491 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pjc2h" podUID="71e540dd-1e1b-46ab-b6d5-01a0ccf6463c" Sep 08 18:37:02.752664 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:37:02.752632 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:37:03.246826 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:37:03.246795 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pjc2h" Sep 08 18:37:07.528778 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:37:07.528741 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:37:07.529155 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:37:07.528852 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 18:37:07.529155 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:37:07.528905 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls podName:71e540dd-1e1b-46ab-b6d5-01a0ccf6463c nodeName:}" failed. No retries permitted until 2026-09-08 18:39:09.528891168 +0000 UTC m=+285.234779378 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls") pod "dns-default-pjc2h" (UID: "71e540dd-1e1b-46ab-b6d5-01a0ccf6463c") : secret "dns-default-metrics-tls" not found Sep 08 18:37:16.739432 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:37:16.739396 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:38:00.378126 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:00.378091 2553 generic.go:356] "Generic (PLEG): container finished" podID="0a006bbd-7ef8-4605-860f-762a7c707723" containerID="73b7b45c4b4f0b2f0dc6c6e483df3e94daca9baa824fb622798b0b416bbc2d6f" exitCode=255 Sep 08 18:38:00.378583 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:00.378150 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq" event={"ID":"0a006bbd-7ef8-4605-860f-762a7c707723","Type":"ContainerDied","Data":"73b7b45c4b4f0b2f0dc6c6e483df3e94daca9baa824fb622798b0b416bbc2d6f"} Sep 08 18:38:00.383512 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:00.383493 2553 scope.go:117] "RemoveContainer" containerID="73b7b45c4b4f0b2f0dc6c6e483df3e94daca9baa824fb622798b0b416bbc2d6f" Sep 08 18:38:01.386201 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:01.386164 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6458cc6948-hktdq" event={"ID":"0a006bbd-7ef8-4605-860f-762a7c707723","Type":"ContainerStarted","Data":"1be1904add41e13b29cfbd11f57f0a8eb89783f6d8740dead1cf2a58e7dc7ac4"} Sep 08 18:38:26.168690 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.168652 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-c6r7f"] Sep 08 18:38:26.170616 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.170598 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-phjbj"] Sep 08 18:38:26.170732 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.170712 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:26.172911 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.172888 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-976857789-j5gl2"] Sep 08 18:38:26.173018 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.173003 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.173459 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.173432 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 08 18:38:26.173705 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.173688 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 08 18:38:26.173854 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.173838 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 08 18:38:26.173927 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.173896 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-48vc2\"" Sep 08 18:38:26.173992 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.173946 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 08 18:38:26.174935 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.174917 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.178660 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.178643 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 08 18:38:26.178861 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.178837 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 08 18:38:26.179016 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.178998 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 08 18:38:26.179102 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.179053 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 08 18:38:26.179244 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.179229 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 08 18:38:26.180011 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.179997 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 08 18:38:26.180341 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.180321 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cvsft\"" Sep 08 18:38:26.180437 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.180339 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xlp48\"" Sep 08 18:38:26.180437 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.180341 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 08 18:38:26.185263 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.185243 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 08 18:38:26.187727 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.187710 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 08 18:38:26.188536 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.188513 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-c6r7f"] Sep 08 18:38:26.189160 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.189144 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-phjbj"] Sep 08 18:38:26.192172 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.191629 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 08 18:38:26.192172 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.192104 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-976857789-j5gl2"] Sep 08 18:38:26.229971 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.229946 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fa729ff-ae68-404f-a6c0-a1c87566f68c-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.230062 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.229973 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd9c567c-3d00-41b2-862a-66c9b85c6fca-ca-trust-extracted\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.230062 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.229993 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acc31f4d-9ae3-4be1-8a1d-415cb8fc6041-config\") pod \"console-operator-85b454874b-c6r7f\" (UID: \"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041\") " pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:26.230062 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.230028 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fa729ff-ae68-404f-a6c0-a1c87566f68c-tmp\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.230062 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.230049 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4fa729ff-ae68-404f-a6c0-a1c87566f68c-snapshots\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.230189 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.230063 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mf85\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-kube-api-access-5mf85\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.230189 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.230104 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd9c567c-3d00-41b2-862a-66c9b85c6fca-trusted-ca\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.230189 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.230130 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd9c567c-3d00-41b2-862a-66c9b85c6fca-installation-pull-secrets\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.230189 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.230150 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-bound-sa-token\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.230189 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.230181 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acc31f4d-9ae3-4be1-8a1d-415cb8fc6041-trusted-ca\") pod \"console-operator-85b454874b-c6r7f\" (UID: \"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041\") " pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:26.230335 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.230232 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/acc31f4d-9ae3-4be1-8a1d-415cb8fc6041-serving-cert\") pod \"console-operator-85b454874b-c6r7f\" (UID: \"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041\") " pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:26.230335 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.230254 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-729cj\" (UniqueName: \"kubernetes.io/projected/acc31f4d-9ae3-4be1-8a1d-415cb8fc6041-kube-api-access-729cj\") pod \"console-operator-85b454874b-c6r7f\" (UID: \"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041\") " pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:26.230335 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.230270 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4fa729ff-ae68-404f-a6c0-a1c87566f68c-serving-cert\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.230335 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.230286 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45g5v\" (UniqueName: \"kubernetes.io/projected/4fa729ff-ae68-404f-a6c0-a1c87566f68c-kube-api-access-45g5v\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.230335 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.230311 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd9c567c-3d00-41b2-862a-66c9b85c6fca-image-registry-private-configuration\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.230335 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.230329 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fa729ff-ae68-404f-a6c0-a1c87566f68c-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.230534 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.230343 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.230534 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.230378 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-certificates\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.256417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.256397 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cgjjn"] Sep 08 18:38:26.258272 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.258258 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cgjjn" Sep 08 18:38:26.260618 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.260601 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 08 18:38:26.260723 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.260606 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 08 18:38:26.260723 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.260629 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-s5x88\"" Sep 08 18:38:26.263763 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.263733 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp"] Sep 08 18:38:26.265708 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.265691 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-59g52"] Sep 08 18:38:26.265863 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.265837 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:26.267720 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.267696 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-59g52" Sep 08 18:38:26.268018 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.267708 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cgjjn"] Sep 08 18:38:26.268808 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.268787 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-k796n\"" Sep 08 18:38:26.269419 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.269400 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 08 18:38:26.269660 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.269645 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 08 18:38:26.271219 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.271185 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 08 18:38:26.271445 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.271423 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 08 18:38:26.271655 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.271639 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7cg8s\"" Sep 08 18:38:26.278660 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.278638 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp"] Sep 08 18:38:26.279418 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.279398 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-59g52"] Sep 08 18:38:26.331478 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331447 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-729cj\" (UniqueName: \"kubernetes.io/projected/acc31f4d-9ae3-4be1-8a1d-415cb8fc6041-kube-api-access-729cj\") pod \"console-operator-85b454874b-c6r7f\" (UID: \"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041\") " pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:26.331596 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331482 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd9c567c-3d00-41b2-862a-66c9b85c6fca-ca-trust-extracted\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.331596 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331507 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acc31f4d-9ae3-4be1-8a1d-415cb8fc6041-trusted-ca\") pod \"console-operator-85b454874b-c6r7f\" (UID: \"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041\") " pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:26.331596 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331537 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkl72\" (UniqueName: \"kubernetes.io/projected/fc960331-f6b2-45c7-b741-9adfacc81a4c-kube-api-access-jkl72\") pod \"network-check-source-685456655f-59g52\" (UID: \"fc960331-f6b2-45c7-b741-9adfacc81a4c\") " pod="openshift-network-diagnostics/network-check-source-685456655f-59g52" Sep 08 18:38:26.331596 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331565 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/acc31f4d-9ae3-4be1-8a1d-415cb8fc6041-serving-cert\") pod \"console-operator-85b454874b-c6r7f\" (UID: \"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041\") " pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:26.331596 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331591 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fa729ff-ae68-404f-a6c0-a1c87566f68c-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.331841 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331612 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.331841 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331641 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-certificates\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.331841 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331704 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd9c567c-3d00-41b2-862a-66c9b85c6fca-trusted-ca\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.331841 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:26.331723 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 18:38:26.331841 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:26.331737 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-976857789-j5gl2: secret "image-registry-tls" not found Sep 08 18:38:26.331841 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331743 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd9c567c-3d00-41b2-862a-66c9b85c6fca-installation-pull-secrets\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.331841 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331786 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fa729ff-ae68-404f-a6c0-a1c87566f68c-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.331841 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:26.331819 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls podName:fd9c567c-3d00-41b2-862a-66c9b85c6fca nodeName:}" failed. No retries permitted until 2026-09-08 18:38:26.831798334 +0000 UTC m=+242.537686549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls") pod "image-registry-976857789-j5gl2" (UID: "fd9c567c-3d00-41b2-862a-66c9b85c6fca") : secret "image-registry-tls" not found Sep 08 18:38:26.332243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331847 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acc31f4d-9ae3-4be1-8a1d-415cb8fc6041-config\") pod \"console-operator-85b454874b-c6r7f\" (UID: \"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041\") " pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:26.332243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331865 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd9c567c-3d00-41b2-862a-66c9b85c6fca-ca-trust-extracted\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.332243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331871 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4fa729ff-ae68-404f-a6c0-a1c87566f68c-snapshots\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.332243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331907 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fa729ff-ae68-404f-a6c0-a1c87566f68c-tmp\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.332243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331933 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mf85\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-kube-api-access-5mf85\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.332243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331961 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-bound-sa-token\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.332243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.331987 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bda1fd47-da9a-49cd-8f40-b44c70207933-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-q5hmp\" (UID: \"bda1fd47-da9a-49cd-8f40-b44c70207933\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:26.332243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.332028 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqj8c\" (UniqueName: \"kubernetes.io/projected/bda1fd47-da9a-49cd-8f40-b44c70207933-kube-api-access-rqj8c\") pod \"cluster-monitoring-operator-6cdcc96965-q5hmp\" (UID: \"bda1fd47-da9a-49cd-8f40-b44c70207933\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:26.332243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.332060 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4fa729ff-ae68-404f-a6c0-a1c87566f68c-serving-cert\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.332243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.332085 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q5hmp\" (UID: \"bda1fd47-da9a-49cd-8f40-b44c70207933\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:26.332243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.332114 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45g5v\" (UniqueName: \"kubernetes.io/projected/4fa729ff-ae68-404f-a6c0-a1c87566f68c-kube-api-access-45g5v\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.332243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.332156 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd9c567c-3d00-41b2-862a-66c9b85c6fca-image-registry-private-configuration\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.332243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.332194 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbfbj\" (UniqueName: \"kubernetes.io/projected/1b6996c8-2d68-429f-a9dd-5b0543a43060-kube-api-access-kbfbj\") pod \"volume-data-source-validator-695c895bd6-cgjjn\" (UID: \"1b6996c8-2d68-429f-a9dd-5b0543a43060\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cgjjn" Sep 08 18:38:26.332902 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.332280 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-certificates\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.332902 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.332327 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fa729ff-ae68-404f-a6c0-a1c87566f68c-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.332902 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.332525 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/acc31f4d-9ae3-4be1-8a1d-415cb8fc6041-trusted-ca\") pod \"console-operator-85b454874b-c6r7f\" (UID: \"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041\") " pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:26.332902 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.332535 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fa729ff-ae68-404f-a6c0-a1c87566f68c-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.332902 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.332843 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/acc31f4d-9ae3-4be1-8a1d-415cb8fc6041-config\") pod \"console-operator-85b454874b-c6r7f\" (UID: \"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041\") " pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:26.332902 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.332886 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4fa729ff-ae68-404f-a6c0-a1c87566f68c-snapshots\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.332902 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.332891 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd9c567c-3d00-41b2-862a-66c9b85c6fca-trusted-ca\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.333232 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.333147 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fa729ff-ae68-404f-a6c0-a1c87566f68c-tmp\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.334199 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.334167 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/acc31f4d-9ae3-4be1-8a1d-415cb8fc6041-serving-cert\") pod \"console-operator-85b454874b-c6r7f\" (UID: \"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041\") " pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:26.334676 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.334657 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd9c567c-3d00-41b2-862a-66c9b85c6fca-installation-pull-secrets\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.334848 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.334827 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4fa729ff-ae68-404f-a6c0-a1c87566f68c-serving-cert\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.334886 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.334848 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd9c567c-3d00-41b2-862a-66c9b85c6fca-image-registry-private-configuration\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.338627 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.338606 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-729cj\" (UniqueName: \"kubernetes.io/projected/acc31f4d-9ae3-4be1-8a1d-415cb8fc6041-kube-api-access-729cj\") pod \"console-operator-85b454874b-c6r7f\" (UID: \"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041\") " pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:26.339921 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.339897 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45g5v\" (UniqueName: \"kubernetes.io/projected/4fa729ff-ae68-404f-a6c0-a1c87566f68c-kube-api-access-45g5v\") pod \"insights-operator-76bcd6ccf5-phjbj\" (UID: \"4fa729ff-ae68-404f-a6c0-a1c87566f68c\") " pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.340784 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.340762 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-bound-sa-token\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.341125 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.341105 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mf85\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-kube-api-access-5mf85\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.433212 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.433151 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkl72\" (UniqueName: \"kubernetes.io/projected/fc960331-f6b2-45c7-b741-9adfacc81a4c-kube-api-access-jkl72\") pod \"network-check-source-685456655f-59g52\" (UID: \"fc960331-f6b2-45c7-b741-9adfacc81a4c\") " pod="openshift-network-diagnostics/network-check-source-685456655f-59g52" Sep 08 18:38:26.433212 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.433208 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bda1fd47-da9a-49cd-8f40-b44c70207933-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-q5hmp\" (UID: \"bda1fd47-da9a-49cd-8f40-b44c70207933\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:26.433408 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.433230 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqj8c\" (UniqueName: \"kubernetes.io/projected/bda1fd47-da9a-49cd-8f40-b44c70207933-kube-api-access-rqj8c\") pod \"cluster-monitoring-operator-6cdcc96965-q5hmp\" (UID: \"bda1fd47-da9a-49cd-8f40-b44c70207933\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:26.433408 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.433342 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q5hmp\" (UID: \"bda1fd47-da9a-49cd-8f40-b44c70207933\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:26.433495 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.433415 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbfbj\" (UniqueName: \"kubernetes.io/projected/1b6996c8-2d68-429f-a9dd-5b0543a43060-kube-api-access-kbfbj\") pod \"volume-data-source-validator-695c895bd6-cgjjn\" (UID: \"1b6996c8-2d68-429f-a9dd-5b0543a43060\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cgjjn" Sep 08 18:38:26.433495 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:26.433462 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 18:38:26.433573 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:26.433535 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls podName:bda1fd47-da9a-49cd-8f40-b44c70207933 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:26.93351558 +0000 UTC m=+242.639403807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q5hmp" (UID: "bda1fd47-da9a-49cd-8f40-b44c70207933") : secret "cluster-monitoring-operator-tls" not found Sep 08 18:38:26.433798 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.433780 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bda1fd47-da9a-49cd-8f40-b44c70207933-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-q5hmp\" (UID: \"bda1fd47-da9a-49cd-8f40-b44c70207933\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:26.441746 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.441724 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkl72\" (UniqueName: \"kubernetes.io/projected/fc960331-f6b2-45c7-b741-9adfacc81a4c-kube-api-access-jkl72\") pod \"network-check-source-685456655f-59g52\" (UID: \"fc960331-f6b2-45c7-b741-9adfacc81a4c\") " pod="openshift-network-diagnostics/network-check-source-685456655f-59g52" Sep 08 18:38:26.441995 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.441977 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbfbj\" (UniqueName: \"kubernetes.io/projected/1b6996c8-2d68-429f-a9dd-5b0543a43060-kube-api-access-kbfbj\") pod \"volume-data-source-validator-695c895bd6-cgjjn\" (UID: \"1b6996c8-2d68-429f-a9dd-5b0543a43060\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cgjjn" Sep 08 18:38:26.442215 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.442188 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqj8c\" (UniqueName: \"kubernetes.io/projected/bda1fd47-da9a-49cd-8f40-b44c70207933-kube-api-access-rqj8c\") pod \"cluster-monitoring-operator-6cdcc96965-q5hmp\" (UID: \"bda1fd47-da9a-49cd-8f40-b44c70207933\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:26.480635 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.480610 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:26.488293 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.488274 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" Sep 08 18:38:26.566799 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.566767 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cgjjn" Sep 08 18:38:26.586994 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.586969 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-59g52" Sep 08 18:38:26.747597 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.747524 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-c6r7f"] Sep 08 18:38:26.751042 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:38:26.751020 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc31f4d_9ae3_4be1_8a1d_415cb8fc6041.slice/crio-c65f535b13e588ed08f9992ad365a72530ba80ddd1660a099cd66a40d9b54216 WatchSource:0}: Error finding container c65f535b13e588ed08f9992ad365a72530ba80ddd1660a099cd66a40d9b54216: Status 404 returned error can't find the container with id c65f535b13e588ed08f9992ad365a72530ba80ddd1660a099cd66a40d9b54216 Sep 08 18:38:26.754034 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.754008 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-phjbj"] Sep 08 18:38:26.756935 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:38:26.756915 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa729ff_ae68_404f_a6c0_a1c87566f68c.slice/crio-9118cd9d3f2265341028a57269ed79501dd3487a7bf54c1b72618750875c5850 WatchSource:0}: Error finding container 9118cd9d3f2265341028a57269ed79501dd3487a7bf54c1b72618750875c5850: Status 404 returned error can't find the container with id 9118cd9d3f2265341028a57269ed79501dd3487a7bf54c1b72618750875c5850 Sep 08 18:38:26.830088 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.830068 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cgjjn"] Sep 08 18:38:26.832139 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:38:26.832105 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6996c8_2d68_429f_a9dd_5b0543a43060.slice/crio-6e77eac1f3048c56f17cdf184bacbe2ec1baa8ec6b1f6571ae265cbaf25239ba WatchSource:0}: Error finding container 6e77eac1f3048c56f17cdf184bacbe2ec1baa8ec6b1f6571ae265cbaf25239ba: Status 404 returned error can't find the container with id 6e77eac1f3048c56f17cdf184bacbe2ec1baa8ec6b1f6571ae265cbaf25239ba Sep 08 18:38:26.836493 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.836468 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:26.836740 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:26.836645 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 18:38:26.836740 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:26.836666 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-976857789-j5gl2: secret "image-registry-tls" not found Sep 08 18:38:26.836740 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:26.836721 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls podName:fd9c567c-3d00-41b2-862a-66c9b85c6fca nodeName:}" failed. No retries permitted until 2026-09-08 18:38:27.83670277 +0000 UTC m=+243.542590986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls") pod "image-registry-976857789-j5gl2" (UID: "fd9c567c-3d00-41b2-862a-66c9b85c6fca") : secret "image-registry-tls" not found Sep 08 18:38:26.853739 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.853669 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-59g52"] Sep 08 18:38:26.855648 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:38:26.855628 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc960331_f6b2_45c7_b741_9adfacc81a4c.slice/crio-8ddda87568523fec4dffdf6d941e86b32e213804ca667cfca53293a14ba82775 WatchSource:0}: Error finding container 8ddda87568523fec4dffdf6d941e86b32e213804ca667cfca53293a14ba82775: Status 404 returned error can't find the container with id 8ddda87568523fec4dffdf6d941e86b32e213804ca667cfca53293a14ba82775 Sep 08 18:38:26.937944 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:26.937913 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q5hmp\" (UID: \"bda1fd47-da9a-49cd-8f40-b44c70207933\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:26.938069 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:26.938020 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 18:38:26.938236 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:26.938071 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls podName:bda1fd47-da9a-49cd-8f40-b44c70207933 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:27.938057588 +0000 UTC m=+243.643945797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q5hmp" (UID: "bda1fd47-da9a-49cd-8f40-b44c70207933") : secret "cluster-monitoring-operator-tls" not found Sep 08 18:38:27.462393 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:27.462306 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cgjjn" event={"ID":"1b6996c8-2d68-429f-a9dd-5b0543a43060","Type":"ContainerStarted","Data":"6e77eac1f3048c56f17cdf184bacbe2ec1baa8ec6b1f6571ae265cbaf25239ba"} Sep 08 18:38:27.465129 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:27.464620 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-59g52" event={"ID":"fc960331-f6b2-45c7-b741-9adfacc81a4c","Type":"ContainerStarted","Data":"7b387544858adc6d4c676f3f303e9fea36e8d386b2d572a03fc098aa518c0996"} Sep 08 18:38:27.465129 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:27.464667 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-59g52" event={"ID":"fc960331-f6b2-45c7-b741-9adfacc81a4c","Type":"ContainerStarted","Data":"8ddda87568523fec4dffdf6d941e86b32e213804ca667cfca53293a14ba82775"} Sep 08 18:38:27.466555 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:27.466487 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" event={"ID":"4fa729ff-ae68-404f-a6c0-a1c87566f68c","Type":"ContainerStarted","Data":"9118cd9d3f2265341028a57269ed79501dd3487a7bf54c1b72618750875c5850"} Sep 08 18:38:27.468194 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:27.468170 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-c6r7f" event={"ID":"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041","Type":"ContainerStarted","Data":"c65f535b13e588ed08f9992ad365a72530ba80ddd1660a099cd66a40d9b54216"} Sep 08 18:38:27.479873 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:27.479477 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-59g52" podStartSLOduration=1.479459898 podStartE2EDuration="1.479459898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:27.477899045 +0000 UTC m=+243.183787274" watchObservedRunningTime="2026-09-08 18:38:27.479459898 +0000 UTC m=+243.185348126" Sep 08 18:38:27.843849 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:27.843767 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:27.844012 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:27.843941 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 18:38:27.844012 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:27.843965 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-976857789-j5gl2: secret "image-registry-tls" not found Sep 08 18:38:27.844114 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:27.844038 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls podName:fd9c567c-3d00-41b2-862a-66c9b85c6fca nodeName:}" failed. No retries permitted until 2026-09-08 18:38:29.844017815 +0000 UTC m=+245.549906120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls") pod "image-registry-976857789-j5gl2" (UID: "fd9c567c-3d00-41b2-862a-66c9b85c6fca") : secret "image-registry-tls" not found Sep 08 18:38:27.944589 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:27.944549 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q5hmp\" (UID: \"bda1fd47-da9a-49cd-8f40-b44c70207933\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:27.944756 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:27.944692 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 18:38:27.944823 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:27.944761 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls podName:bda1fd47-da9a-49cd-8f40-b44c70207933 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:29.944740459 +0000 UTC m=+245.650628691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q5hmp" (UID: "bda1fd47-da9a-49cd-8f40-b44c70207933") : secret "cluster-monitoring-operator-tls" not found Sep 08 18:38:28.474202 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:28.474111 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cgjjn" event={"ID":"1b6996c8-2d68-429f-a9dd-5b0543a43060","Type":"ContainerStarted","Data":"ab0537ef4d282c0667e64a3d340e29a98464f47b9c2d4c4bfdca0ec4a47ec5ef"} Sep 08 18:38:28.489088 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:28.489035 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cgjjn" podStartSLOduration=1.142989217 podStartE2EDuration="2.489014961s" totalImagesPullingTime="1.346025744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:28.487883277 +0000 UTC m=+244.193771512" watchObservedRunningTime="2026-09-08 18:38:28.489014961 +0000 UTC m=+244.194903199" Sep 08 18:38:29.478537 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:29.478504 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" event={"ID":"4fa729ff-ae68-404f-a6c0-a1c87566f68c","Type":"ContainerStarted","Data":"b02ec9e8907dcaef374ad202752d873aa95b20f21ebc2635ed413f45728faf9c"} Sep 08 18:38:29.480212 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:29.480191 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/0.log" Sep 08 18:38:29.480321 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:29.480227 2553 generic.go:356] "Generic (PLEG): container finished" podID="acc31f4d-9ae3-4be1-8a1d-415cb8fc6041" containerID="c35d62cbc9f289602c11b97260a6743e3b73268767377a9139b21eb77bb77cb4" exitCode=255 Sep 08 18:38:29.480321 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:29.480257 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-c6r7f" event={"ID":"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041","Type":"ContainerDied","Data":"c35d62cbc9f289602c11b97260a6743e3b73268767377a9139b21eb77bb77cb4"} Sep 08 18:38:29.480478 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:29.480464 2553 scope.go:117] "RemoveContainer" containerID="c35d62cbc9f289602c11b97260a6743e3b73268767377a9139b21eb77bb77cb4" Sep 08 18:38:29.513496 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:29.513439 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" podStartSLOduration=0.9995573 podStartE2EDuration="3.513420988s" totalImagesPullingTime="2.513863688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:29.496912742 +0000 UTC m=+245.202800968" watchObservedRunningTime="2026-09-08 18:38:29.513420988 +0000 UTC m=+245.219309220" Sep 08 18:38:29.858604 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:29.858522 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:29.858734 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:29.858653 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 18:38:29.858734 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:29.858671 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-976857789-j5gl2: secret "image-registry-tls" not found Sep 08 18:38:29.858809 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:29.858750 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls podName:fd9c567c-3d00-41b2-862a-66c9b85c6fca nodeName:}" failed. No retries permitted until 2026-09-08 18:38:33.858734644 +0000 UTC m=+249.564622854 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls") pod "image-registry-976857789-j5gl2" (UID: "fd9c567c-3d00-41b2-862a-66c9b85c6fca") : secret "image-registry-tls" not found Sep 08 18:38:29.958843 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:29.958810 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q5hmp\" (UID: \"bda1fd47-da9a-49cd-8f40-b44c70207933\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:29.958979 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:29.958924 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 18:38:29.959015 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:29.958993 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls podName:bda1fd47-da9a-49cd-8f40-b44c70207933 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:33.958977771 +0000 UTC m=+249.664866002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q5hmp" (UID: "bda1fd47-da9a-49cd-8f40-b44c70207933") : secret "cluster-monitoring-operator-tls" not found Sep 08 18:38:30.484855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.484824 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 18:38:30.485265 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.485250 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/0.log" Sep 08 18:38:30.485306 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.485284 2553 generic.go:356] "Generic (PLEG): container finished" podID="acc31f4d-9ae3-4be1-8a1d-415cb8fc6041" containerID="f77be5de62c2b503d2b33562520a64f015098ae486de77a81ce557347f44c361" exitCode=255 Sep 08 18:38:30.485336 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.485314 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-c6r7f" event={"ID":"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041","Type":"ContainerDied","Data":"f77be5de62c2b503d2b33562520a64f015098ae486de77a81ce557347f44c361"} Sep 08 18:38:30.485409 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.485396 2553 scope.go:117] "RemoveContainer" containerID="c35d62cbc9f289602c11b97260a6743e3b73268767377a9139b21eb77bb77cb4" Sep 08 18:38:30.485623 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.485601 2553 scope.go:117] "RemoveContainer" containerID="f77be5de62c2b503d2b33562520a64f015098ae486de77a81ce557347f44c361" Sep 08 18:38:30.485798 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:30.485779 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-c6r7f_openshift-console-operator(acc31f4d-9ae3-4be1-8a1d-415cb8fc6041)\"" pod="openshift-console-operator/console-operator-85b454874b-c6r7f" podUID="acc31f4d-9ae3-4be1-8a1d-415cb8fc6041" Sep 08 18:38:30.945042 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.945011 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nclnr"] Sep 08 18:38:30.947722 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.947706 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:30.950337 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.950315 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 08 18:38:30.950337 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.950334 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 08 18:38:30.950510 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.950340 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q24t4\"" Sep 08 18:38:30.956749 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.956728 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nclnr"] Sep 08 18:38:30.967201 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.967179 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:30.967303 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.967221 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:30.967303 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.967241 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-crio-socket\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:30.967303 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.967268 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-data-volume\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:30.967303 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:30.967295 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx79l\" (UniqueName: \"kubernetes.io/projected/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-api-access-qx79l\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:31.068037 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.068010 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:31.068172 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.068046 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-crio-socket\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:31.068172 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.068081 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-data-volume\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:31.068172 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.068156 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-crio-socket\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:31.068267 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:31.068168 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 18:38:31.068267 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:31.068236 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls podName:bcb2c470-bbc4-4b9d-8e81-ad048bb0c693 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:31.568217101 +0000 UTC m=+247.274105326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nclnr" (UID: "bcb2c470-bbc4-4b9d-8e81-ad048bb0c693") : secret "insights-runtime-extractor-tls" not found Sep 08 18:38:31.068267 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.068252 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx79l\" (UniqueName: \"kubernetes.io/projected/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-api-access-qx79l\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:31.068384 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.068330 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:31.068384 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.068374 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-data-volume\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:31.068483 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:31.068466 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 18:38:31.068548 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:31.068533 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm podName:bcb2c470-bbc4-4b9d-8e81-ad048bb0c693 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:31.568514227 +0000 UTC m=+247.274402456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nclnr" (UID: "bcb2c470-bbc4-4b9d-8e81-ad048bb0c693") : configmap "kube-rbac-proxy" not found Sep 08 18:38:31.077488 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.077469 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx79l\" (UniqueName: \"kubernetes.io/projected/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-api-access-qx79l\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:31.489923 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.489898 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 18:38:31.490277 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.490187 2553 scope.go:117] "RemoveContainer" containerID="f77be5de62c2b503d2b33562520a64f015098ae486de77a81ce557347f44c361" Sep 08 18:38:31.490349 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:31.490329 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-c6r7f_openshift-console-operator(acc31f4d-9ae3-4be1-8a1d-415cb8fc6041)\"" pod="openshift-console-operator/console-operator-85b454874b-c6r7f" podUID="acc31f4d-9ae3-4be1-8a1d-415cb8fc6041" Sep 08 18:38:31.572484 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.572456 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:31.572620 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:31.572535 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 18:38:31.572620 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:31.572584 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm podName:bcb2c470-bbc4-4b9d-8e81-ad048bb0c693 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:32.572561914 +0000 UTC m=+248.278450137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nclnr" (UID: "bcb2c470-bbc4-4b9d-8e81-ad048bb0c693") : configmap "kube-rbac-proxy" not found Sep 08 18:38:31.572620 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.572610 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:31.572748 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:31.572718 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 18:38:31.572791 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:31.572784 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls podName:bcb2c470-bbc4-4b9d-8e81-ad048bb0c693 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:32.572769523 +0000 UTC m=+248.278657750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nclnr" (UID: "bcb2c470-bbc4-4b9d-8e81-ad048bb0c693") : secret "insights-runtime-extractor-tls" not found Sep 08 18:38:31.658792 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.658765 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fbzzv"] Sep 08 18:38:31.660951 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.660936 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fbzzv" Sep 08 18:38:31.663187 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.663171 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 08 18:38:31.663588 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.663573 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 08 18:38:31.663638 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.663573 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rbsfh\"" Sep 08 18:38:31.669893 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.669874 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fbzzv"] Sep 08 18:38:31.773492 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.773434 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmskg\" (UniqueName: \"kubernetes.io/projected/77e317bc-d870-4bb8-a823-6a240c26a5ff-kube-api-access-bmskg\") pod \"migrator-fdbb8fdff-fbzzv\" (UID: \"77e317bc-d870-4bb8-a823-6a240c26a5ff\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fbzzv" Sep 08 18:38:31.874346 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.874316 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmskg\" (UniqueName: \"kubernetes.io/projected/77e317bc-d870-4bb8-a823-6a240c26a5ff-kube-api-access-bmskg\") pod \"migrator-fdbb8fdff-fbzzv\" (UID: \"77e317bc-d870-4bb8-a823-6a240c26a5ff\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fbzzv" Sep 08 18:38:31.881691 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.881661 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmskg\" (UniqueName: \"kubernetes.io/projected/77e317bc-d870-4bb8-a823-6a240c26a5ff-kube-api-access-bmskg\") pod \"migrator-fdbb8fdff-fbzzv\" (UID: \"77e317bc-d870-4bb8-a823-6a240c26a5ff\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fbzzv" Sep 08 18:38:31.969790 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:31.969759 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fbzzv" Sep 08 18:38:32.233867 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:32.233691 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fbzzv"] Sep 08 18:38:32.236290 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:38:32.236265 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e317bc_d870_4bb8_a823_6a240c26a5ff.slice/crio-c9b4210769a5021bb6b4f03baf8025ebde64e6e967e00aa0559557205c1ef9cc WatchSource:0}: Error finding container c9b4210769a5021bb6b4f03baf8025ebde64e6e967e00aa0559557205c1ef9cc: Status 404 returned error can't find the container with id c9b4210769a5021bb6b4f03baf8025ebde64e6e967e00aa0559557205c1ef9cc Sep 08 18:38:32.495461 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:32.494093 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fbzzv" event={"ID":"77e317bc-d870-4bb8-a823-6a240c26a5ff","Type":"ContainerStarted","Data":"c9b4210769a5021bb6b4f03baf8025ebde64e6e967e00aa0559557205c1ef9cc"} Sep 08 18:38:32.579832 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:32.579786 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:32.579999 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:32.579856 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:32.579999 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:32.579900 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 18:38:32.579999 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:32.579952 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm podName:bcb2c470-bbc4-4b9d-8e81-ad048bb0c693 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:34.579938963 +0000 UTC m=+250.285827178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nclnr" (UID: "bcb2c470-bbc4-4b9d-8e81-ad048bb0c693") : configmap "kube-rbac-proxy" not found Sep 08 18:38:32.580113 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:32.579997 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 18:38:32.580113 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:32.580057 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls podName:bcb2c470-bbc4-4b9d-8e81-ad048bb0c693 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:34.580042589 +0000 UTC m=+250.285930817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nclnr" (UID: "bcb2c470-bbc4-4b9d-8e81-ad048bb0c693") : secret "insights-runtime-extractor-tls" not found Sep 08 18:38:33.105783 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:33.105758 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xmpj8_96b33d5d-d099-464a-adcc-03562ae049c0/dns-node-resolver/0.log" Sep 08 18:38:33.499718 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:33.499687 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fbzzv" event={"ID":"77e317bc-d870-4bb8-a823-6a240c26a5ff","Type":"ContainerStarted","Data":"d36ab6ad2d67a2a5849086a0f2dbe6ed764d2e1192ab518d610e231c83d8d3a4"} Sep 08 18:38:33.499989 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:33.499780 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fbzzv" event={"ID":"77e317bc-d870-4bb8-a823-6a240c26a5ff","Type":"ContainerStarted","Data":"55314b28073ddd6cb2df7b31c7f5b68291b43aab6dfa56f3180b3a40a4dedaa1"} Sep 08 18:38:33.514786 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:33.514750 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fbzzv" podStartSLOduration=1.421367166 podStartE2EDuration="2.514737568s" totalImagesPullingTime="1.093370402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:33.51344632 +0000 UTC m=+249.219334553" watchObservedRunningTime="2026-09-08 18:38:33.514737568 +0000 UTC m=+249.220625799" Sep 08 18:38:33.889493 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:33.889461 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:33.889655 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:33.889613 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 18:38:33.889655 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:33.889634 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-976857789-j5gl2: secret "image-registry-tls" not found Sep 08 18:38:33.889727 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:33.889687 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls podName:fd9c567c-3d00-41b2-862a-66c9b85c6fca nodeName:}" failed. No retries permitted until 2026-09-08 18:38:41.88967182 +0000 UTC m=+257.595560045 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls") pod "image-registry-976857789-j5gl2" (UID: "fd9c567c-3d00-41b2-862a-66c9b85c6fca") : secret "image-registry-tls" not found Sep 08 18:38:33.905759 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:33.905736 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jq7qc_1911e404-7356-4fd2-86af-acc9012c5a4b/node-ca/0.log" Sep 08 18:38:33.990612 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:33.990575 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q5hmp\" (UID: \"bda1fd47-da9a-49cd-8f40-b44c70207933\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:33.990760 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:33.990713 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 18:38:33.990841 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:33.990769 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls podName:bda1fd47-da9a-49cd-8f40-b44c70207933 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:41.990754926 +0000 UTC m=+257.696643141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q5hmp" (UID: "bda1fd47-da9a-49cd-8f40-b44c70207933") : secret "cluster-monitoring-operator-tls" not found Sep 08 18:38:34.595350 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:34.595316 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:34.595721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:34.595377 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:34.595721 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:34.595470 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 18:38:34.595721 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:34.595471 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 18:38:34.595721 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:34.595518 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls podName:bcb2c470-bbc4-4b9d-8e81-ad048bb0c693 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:38.595506507 +0000 UTC m=+254.301394717 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nclnr" (UID: "bcb2c470-bbc4-4b9d-8e81-ad048bb0c693") : secret "insights-runtime-extractor-tls" not found Sep 08 18:38:34.595721 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:34.595531 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm podName:bcb2c470-bbc4-4b9d-8e81-ad048bb0c693 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:38.59552549 +0000 UTC m=+254.301413699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nclnr" (UID: "bcb2c470-bbc4-4b9d-8e81-ad048bb0c693") : configmap "kube-rbac-proxy" not found Sep 08 18:38:34.905687 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:34.905639 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mnblg_2ae8b37f-62ce-4a27-ae4b-369e38c6de40/serve-healthcheck-canary/0.log" Sep 08 18:38:36.481329 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:36.481250 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:36.481329 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:36.481301 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:36.481701 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:36.481616 2553 scope.go:117] "RemoveContainer" containerID="f77be5de62c2b503d2b33562520a64f015098ae486de77a81ce557347f44c361" Sep 08 18:38:36.481769 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:36.481751 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-c6r7f_openshift-console-operator(acc31f4d-9ae3-4be1-8a1d-415cb8fc6041)\"" pod="openshift-console-operator/console-operator-85b454874b-c6r7f" podUID="acc31f4d-9ae3-4be1-8a1d-415cb8fc6041" Sep 08 18:38:36.609690 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:36.609663 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:38:36.609836 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:36.609776 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 08 18:38:36.609836 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:36.609827 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs podName:40b6e42c-6d96-445d-8b83-108320730adb nodeName:}" failed. No retries permitted until 2026-09-08 18:40:38.609813147 +0000 UTC m=+374.315701356 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs") pod "network-metrics-daemon-sjrbw" (UID: "40b6e42c-6d96-445d-8b83-108320730adb") : secret "metrics-daemon-secret" not found Sep 08 18:38:38.628118 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:38.628078 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:38.628558 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:38.628132 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:38.628558 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:38.628223 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 18:38:38.628558 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:38.628258 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 18:38:38.628558 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:38.628321 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm podName:bcb2c470-bbc4-4b9d-8e81-ad048bb0c693 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:46.628299086 +0000 UTC m=+262.334187296 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nclnr" (UID: "bcb2c470-bbc4-4b9d-8e81-ad048bb0c693") : configmap "kube-rbac-proxy" not found Sep 08 18:38:38.628558 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:38.628344 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls podName:bcb2c470-bbc4-4b9d-8e81-ad048bb0c693 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:46.628333633 +0000 UTC m=+262.334221845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nclnr" (UID: "bcb2c470-bbc4-4b9d-8e81-ad048bb0c693") : secret "insights-runtime-extractor-tls" not found Sep 08 18:38:41.955119 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:41.955071 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:41.957409 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:41.957389 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls\") pod \"image-registry-976857789-j5gl2\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:42.055714 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:42.055679 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q5hmp\" (UID: \"bda1fd47-da9a-49cd-8f40-b44c70207933\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:42.055852 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:42.055819 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 18:38:42.055899 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:38:42.055883 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls podName:bda1fd47-da9a-49cd-8f40-b44c70207933 nodeName:}" failed. No retries permitted until 2026-09-08 18:38:58.05586732 +0000 UTC m=+273.761755535 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q5hmp" (UID: "bda1fd47-da9a-49cd-8f40-b44c70207933") : secret "cluster-monitoring-operator-tls" not found Sep 08 18:38:42.094855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:42.094831 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:42.368247 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:42.368170 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-976857789-j5gl2"] Sep 08 18:38:42.371376 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:38:42.371334 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9c567c_3d00_41b2_862a_66c9b85c6fca.slice/crio-29d6f684dca25af313fd95d4d977d55dc4a8a9c510549b6f26dc765325f8decf WatchSource:0}: Error finding container 29d6f684dca25af313fd95d4d977d55dc4a8a9c510549b6f26dc765325f8decf: Status 404 returned error can't find the container with id 29d6f684dca25af313fd95d4d977d55dc4a8a9c510549b6f26dc765325f8decf Sep 08 18:38:42.527708 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:42.527666 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-976857789-j5gl2" event={"ID":"fd9c567c-3d00-41b2-862a-66c9b85c6fca","Type":"ContainerStarted","Data":"a06b943e8d97428462b2b452f4176ff65d2dfdae803e569f8b9762795174cc43"} Sep 08 18:38:42.527863 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:42.527723 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-976857789-j5gl2" event={"ID":"fd9c567c-3d00-41b2-862a-66c9b85c6fca","Type":"ContainerStarted","Data":"29d6f684dca25af313fd95d4d977d55dc4a8a9c510549b6f26dc765325f8decf"} Sep 08 18:38:42.527863 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:42.527755 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:38:42.544882 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:42.544831 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-976857789-j5gl2" podStartSLOduration=16.544816047 podStartE2EDuration="16.544816047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:42.543909842 +0000 UTC m=+258.249798076" watchObservedRunningTime="2026-09-08 18:38:42.544816047 +0000 UTC m=+258.250704280" Sep 08 18:38:46.689887 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:46.689853 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:46.690246 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:46.689906 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:46.690543 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:46.690516 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:46.692212 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:46.692194 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcb2c470-bbc4-4b9d-8e81-ad048bb0c693-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nclnr\" (UID: \"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693\") " pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:46.856386 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:46.856323 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nclnr" Sep 08 18:38:47.125594 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:47.125564 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nclnr"] Sep 08 18:38:47.128295 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:38:47.128270 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb2c470_bbc4_4b9d_8e81_ad048bb0c693.slice/crio-6a5855a4f1632abe6b67b052c20d5dfe008293640068970a9fbeceece17e442c WatchSource:0}: Error finding container 6a5855a4f1632abe6b67b052c20d5dfe008293640068970a9fbeceece17e442c: Status 404 returned error can't find the container with id 6a5855a4f1632abe6b67b052c20d5dfe008293640068970a9fbeceece17e442c Sep 08 18:38:47.544302 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:47.544220 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nclnr" event={"ID":"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693","Type":"ContainerStarted","Data":"8ebac49f3fe7242c23f1ed14c95a03cab45638657442426662891162e487decd"} Sep 08 18:38:47.544302 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:47.544283 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nclnr" event={"ID":"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693","Type":"ContainerStarted","Data":"6a5855a4f1632abe6b67b052c20d5dfe008293640068970a9fbeceece17e442c"} Sep 08 18:38:48.549253 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:48.549219 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nclnr" event={"ID":"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693","Type":"ContainerStarted","Data":"613eb03c3297005239502bc4d20affc7b93600e2743d177535e9a4ae76c07a55"} Sep 08 18:38:49.739708 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:49.739686 2553 scope.go:117] "RemoveContainer" containerID="f77be5de62c2b503d2b33562520a64f015098ae486de77a81ce557347f44c361" Sep 08 18:38:50.557760 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:50.557728 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 18:38:50.557930 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:50.557825 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-c6r7f" event={"ID":"acc31f4d-9ae3-4be1-8a1d-415cb8fc6041","Type":"ContainerStarted","Data":"d948331dfb7cbfb660d221f38f2fc11b05a2aa027402ebbc78ab6e579ea1ebb0"} Sep 08 18:38:50.558181 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:50.558154 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:50.559988 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:50.559965 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nclnr" event={"ID":"bcb2c470-bbc4-4b9d-8e81-ad048bb0c693","Type":"ContainerStarted","Data":"0ed6744aebbb092138057fa28f515d29c61e99cf424f513d0175601f875f19f3"} Sep 08 18:38:50.563204 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:50.563188 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-c6r7f" Sep 08 18:38:50.574518 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:50.574482 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-c6r7f" podStartSLOduration=22.053408505 podStartE2EDuration="24.574469835s" totalImagesPullingTime="2.52106133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:50.573897736 +0000 UTC m=+266.279785979" watchObservedRunningTime="2026-09-08 18:38:50.574469835 +0000 UTC m=+266.280358084" Sep 08 18:38:50.604438 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:50.604397 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nclnr" podStartSLOduration=18.226089607 podStartE2EDuration="20.604384994s" totalImagesPullingTime="2.378295387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 18:38:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:50.603555929 +0000 UTC m=+266.309444161" watchObservedRunningTime="2026-09-08 18:38:50.604384994 +0000 UTC m=+266.310273217" Sep 08 18:38:52.011088 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.011059 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-976857789-j5gl2"] Sep 08 18:38:52.052054 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.052021 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55c57bf558-n9hfq"] Sep 08 18:38:52.054862 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.054847 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.066323 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.066297 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55c57bf558-n9hfq"] Sep 08 18:38:52.138970 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.138945 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3d3761a-ecc1-4045-bf09-80137816248f-ca-trust-extracted\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.139109 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.138980 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3d3761a-ecc1-4045-bf09-80137816248f-registry-tls\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.139109 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.139048 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3d3761a-ecc1-4045-bf09-80137816248f-registry-certificates\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.139109 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.139089 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3d3761a-ecc1-4045-bf09-80137816248f-image-registry-private-configuration\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.139211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.139110 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3d3761a-ecc1-4045-bf09-80137816248f-trusted-ca\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.139211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.139130 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3d3761a-ecc1-4045-bf09-80137816248f-bound-sa-token\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.139211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.139185 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2mtb\" (UniqueName: \"kubernetes.io/projected/d3d3761a-ecc1-4045-bf09-80137816248f-kube-api-access-h2mtb\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.139298 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.139214 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3d3761a-ecc1-4045-bf09-80137816248f-installation-pull-secrets\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.239716 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.239684 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3d3761a-ecc1-4045-bf09-80137816248f-ca-trust-extracted\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.239716 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.239716 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3d3761a-ecc1-4045-bf09-80137816248f-registry-tls\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.239907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.239747 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3d3761a-ecc1-4045-bf09-80137816248f-registry-certificates\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.239907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.239769 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3d3761a-ecc1-4045-bf09-80137816248f-image-registry-private-configuration\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.239907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.239788 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3d3761a-ecc1-4045-bf09-80137816248f-trusted-ca\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.239907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.239808 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3d3761a-ecc1-4045-bf09-80137816248f-bound-sa-token\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.239907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.239825 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2mtb\" (UniqueName: \"kubernetes.io/projected/d3d3761a-ecc1-4045-bf09-80137816248f-kube-api-access-h2mtb\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.239907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.239846 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3d3761a-ecc1-4045-bf09-80137816248f-installation-pull-secrets\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.240228 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.240162 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d3d3761a-ecc1-4045-bf09-80137816248f-ca-trust-extracted\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.240673 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.240647 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d3d3761a-ecc1-4045-bf09-80137816248f-registry-certificates\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.240791 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.240771 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d3d3761a-ecc1-4045-bf09-80137816248f-trusted-ca\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.242263 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.242240 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d3d3761a-ecc1-4045-bf09-80137816248f-image-registry-private-configuration\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.242346 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.242247 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d3d3761a-ecc1-4045-bf09-80137816248f-registry-tls\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.242346 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.242294 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d3d3761a-ecc1-4045-bf09-80137816248f-installation-pull-secrets\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.246762 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.246743 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d3d3761a-ecc1-4045-bf09-80137816248f-bound-sa-token\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.247025 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.247006 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2mtb\" (UniqueName: \"kubernetes.io/projected/d3d3761a-ecc1-4045-bf09-80137816248f-kube-api-access-h2mtb\") pod \"image-registry-55c57bf558-n9hfq\" (UID: \"d3d3761a-ecc1-4045-bf09-80137816248f\") " pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.363192 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.363124 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:52.631833 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:52.631805 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55c57bf558-n9hfq"] Sep 08 18:38:52.634589 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:38:52.634564 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d3761a_ecc1_4045_bf09_80137816248f.slice/crio-df4d4a85a5534c0a8960a9970852ff80b3b2b924a4da985aba82f0687f2db8d1 WatchSource:0}: Error finding container df4d4a85a5534c0a8960a9970852ff80b3b2b924a4da985aba82f0687f2db8d1: Status 404 returned error can't find the container with id df4d4a85a5534c0a8960a9970852ff80b3b2b924a4da985aba82f0687f2db8d1 Sep 08 18:38:53.570982 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:53.570944 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" event={"ID":"d3d3761a-ecc1-4045-bf09-80137816248f","Type":"ContainerStarted","Data":"f4a82ebf61f00bc44e8b44ab655c16dde9ac5890b2ce9f4239052faea6ed5705"} Sep 08 18:38:53.571333 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:53.570992 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" event={"ID":"d3d3761a-ecc1-4045-bf09-80137816248f","Type":"ContainerStarted","Data":"df4d4a85a5534c0a8960a9970852ff80b3b2b924a4da985aba82f0687f2db8d1"} Sep 08 18:38:53.571333 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:53.571054 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:38:53.590192 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:53.590148 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" podStartSLOduration=1.590134606 podStartE2EDuration="1.590134606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:38:53.589058035 +0000 UTC m=+269.294946266" watchObservedRunningTime="2026-09-08 18:38:53.590134606 +0000 UTC m=+269.296022837" Sep 08 18:38:58.088057 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:58.088014 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q5hmp\" (UID: \"bda1fd47-da9a-49cd-8f40-b44c70207933\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:58.090356 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:58.090328 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bda1fd47-da9a-49cd-8f40-b44c70207933-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q5hmp\" (UID: \"bda1fd47-da9a-49cd-8f40-b44c70207933\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:58.381658 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:58.381631 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" Sep 08 18:38:58.648095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:58.648068 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp"] Sep 08 18:38:58.650336 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:38:58.650306 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda1fd47_da9a_49cd_8f40_b44c70207933.slice/crio-e6b4479bf92df1f9108b1f5c2c74c2d2ede1fa286dbbda814826ad291809884c WatchSource:0}: Error finding container e6b4479bf92df1f9108b1f5c2c74c2d2ede1fa286dbbda814826ad291809884c: Status 404 returned error can't find the container with id e6b4479bf92df1f9108b1f5c2c74c2d2ede1fa286dbbda814826ad291809884c Sep 08 18:38:59.594911 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:38:59.594876 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" event={"ID":"bda1fd47-da9a-49cd-8f40-b44c70207933","Type":"ContainerStarted","Data":"e6b4479bf92df1f9108b1f5c2c74c2d2ede1fa286dbbda814826ad291809884c"} Sep 08 18:39:01.602527 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:01.602492 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" event={"ID":"bda1fd47-da9a-49cd-8f40-b44c70207933","Type":"ContainerStarted","Data":"52604e317b07756bec207139ed6b60c920e77858494089ebf05220f01a115e37"} Sep 08 18:39:01.617203 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:01.617153 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q5hmp" podStartSLOduration=33.640943141 podStartE2EDuration="35.617137568s" totalImagesPullingTime="1.976194427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:01.616149106 +0000 UTC m=+277.322037338" watchObservedRunningTime="2026-09-08 18:39:01.617137568 +0000 UTC m=+277.323025863" Sep 08 18:39:02.017247 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:02.017213 2553 patch_prober.go:28] interesting pod/image-registry-976857789-j5gl2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 08 18:39:02.017433 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:02.017261 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-976857789-j5gl2" podUID="fd9c567c-3d00-41b2-862a-66c9b85c6fca" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 18:39:06.247312 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:39:06.247263 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pjc2h" podUID="71e540dd-1e1b-46ab-b6d5-01a0ccf6463c" Sep 08 18:39:06.620779 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:06.620703 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pjc2h" Sep 08 18:39:09.476651 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.476614 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc"] Sep 08 18:39:09.479406 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.479389 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.482184 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.482161 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 08 18:39:09.482324 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.482169 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pzfjw\"" Sep 08 18:39:09.483388 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.483350 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 08 18:39:09.483487 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.483398 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 08 18:39:09.483487 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.483424 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 08 18:39:09.490092 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.490070 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc"] Sep 08 18:39:09.493189 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.493169 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2htzd"] Sep 08 18:39:09.495478 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.495461 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.498255 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.498234 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 08 18:39:09.498377 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.498242 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rfb7z\"" Sep 08 18:39:09.498377 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.498252 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 08 18:39:09.498377 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.498279 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 08 18:39:09.573428 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.573393 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f3594cff-d79f-4fec-97eb-baee86351196-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.573602 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.573438 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3594cff-d79f-4fec-97eb-baee86351196-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.573602 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.573516 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f3594cff-d79f-4fec-97eb-baee86351196-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.573602 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.573571 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3594cff-d79f-4fec-97eb-baee86351196-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.573708 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.573622 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:39:09.573708 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.573652 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3594cff-d79f-4fec-97eb-baee86351196-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.573708 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.573701 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbg4g\" (UniqueName: \"kubernetes.io/projected/f3594cff-d79f-4fec-97eb-baee86351196-kube-api-access-kbg4g\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.575855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.575832 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/71e540dd-1e1b-46ab-b6d5-01a0ccf6463c-metrics-tls\") pod \"dns-default-pjc2h\" (UID: \"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c\") " pod="openshift-dns/dns-default-pjc2h" Sep 08 18:39:09.624217 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.624192 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rsh8d\"" Sep 08 18:39:09.632197 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.632172 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pjc2h" Sep 08 18:39:09.674224 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.674184 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbr6l\" (UniqueName: \"kubernetes.io/projected/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-kube-api-access-dbr6l\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.674224 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.674226 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3594cff-d79f-4fec-97eb-baee86351196-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.674453 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.674247 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-sys\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.674453 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.674262 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-tls\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.674453 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.674284 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-textfile\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.674453 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.674300 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.674453 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.674321 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbg4g\" (UniqueName: \"kubernetes.io/projected/f3594cff-d79f-4fec-97eb-baee86351196-kube-api-access-kbg4g\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.674453 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.674342 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-root\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.674453 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.674383 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f3594cff-d79f-4fec-97eb-baee86351196-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.674453 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.674406 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-wtmp\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.674453 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.674423 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.674453 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.674444 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3594cff-d79f-4fec-97eb-baee86351196-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.674918 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.674471 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-metrics-client-ca\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.674918 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.674496 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f3594cff-d79f-4fec-97eb-baee86351196-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.674918 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.674515 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3594cff-d79f-4fec-97eb-baee86351196-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.674918 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:39:09.674640 2553 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 08 18:39:09.674918 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:39:09.674701 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3594cff-d79f-4fec-97eb-baee86351196-kube-state-metrics-tls podName:f3594cff-d79f-4fec-97eb-baee86351196 nodeName:}" failed. No retries permitted until 2026-09-08 18:39:10.174682202 +0000 UTC m=+285.880570426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f3594cff-d79f-4fec-97eb-baee86351196-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-gd5qc" (UID: "f3594cff-d79f-4fec-97eb-baee86351196") : secret "kube-state-metrics-tls" not found Sep 08 18:39:09.676540 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.675696 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f3594cff-d79f-4fec-97eb-baee86351196-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.676540 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.675774 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f3594cff-d79f-4fec-97eb-baee86351196-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.676540 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.675859 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3594cff-d79f-4fec-97eb-baee86351196-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.678541 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.678297 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3594cff-d79f-4fec-97eb-baee86351196-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.685092 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.685068 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbg4g\" (UniqueName: \"kubernetes.io/projected/f3594cff-d79f-4fec-97eb-baee86351196-kube-api-access-kbg4g\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:09.775166 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.775092 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-sys\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.775166 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.775124 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-tls\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.775166 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.775147 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-textfile\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.775433 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.775174 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.775433 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.775206 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-root\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.775433 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.775208 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-sys\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.775433 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.775231 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-wtmp\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.775433 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:39:09.775257 2553 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 08 18:39:09.775433 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.775272 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.775433 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:39:09.775315 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-tls podName:c8ede2fd-b04a-4bfe-be85-b74a047eaf5d nodeName:}" failed. No retries permitted until 2026-09-08 18:39:10.275296032 +0000 UTC m=+285.981184251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-tls") pod "node-exporter-2htzd" (UID: "c8ede2fd-b04a-4bfe-be85-b74a047eaf5d") : secret "node-exporter-tls" not found Sep 08 18:39:09.775433 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.775343 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-wtmp\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.775433 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.775346 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-metrics-client-ca\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.775772 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.775467 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbr6l\" (UniqueName: \"kubernetes.io/projected/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-kube-api-access-dbr6l\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.775772 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.775344 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-root\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.775772 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.775590 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-textfile\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.775872 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.775810 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.775949 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.775931 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-metrics-client-ca\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.777403 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.777383 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.783783 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.783763 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbr6l\" (UniqueName: \"kubernetes.io/projected/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-kube-api-access-dbr6l\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:09.899375 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:09.899333 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pjc2h"] Sep 08 18:39:09.901986 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:39:09.901957 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e540dd_1e1b_46ab_b6d5_01a0ccf6463c.slice/crio-394ff641751bc6509ddcf705554cf158921954ce9310214359f76e3a3ef96d93 WatchSource:0}: Error finding container 394ff641751bc6509ddcf705554cf158921954ce9310214359f76e3a3ef96d93: Status 404 returned error can't find the container with id 394ff641751bc6509ddcf705554cf158921954ce9310214359f76e3a3ef96d93 Sep 08 18:39:10.178693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.178659 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3594cff-d79f-4fec-97eb-baee86351196-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:10.181456 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.181428 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f3594cff-d79f-4fec-97eb-baee86351196-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gd5qc\" (UID: \"f3594cff-d79f-4fec-97eb-baee86351196\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:10.280095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.280062 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-tls\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:10.282292 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.282267 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c8ede2fd-b04a-4bfe-be85-b74a047eaf5d-node-exporter-tls\") pod \"node-exporter-2htzd\" (UID: \"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d\") " pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:10.388010 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.387976 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" Sep 08 18:39:10.404519 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.404491 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2htzd" Sep 08 18:39:10.635232 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.635076 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2htzd" event={"ID":"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d","Type":"ContainerStarted","Data":"7dc7822ed391655d443ac3a5076f948827368420535ed70622c68f78aec036ae"} Sep 08 18:39:10.636604 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.636578 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pjc2h" event={"ID":"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c","Type":"ContainerStarted","Data":"394ff641751bc6509ddcf705554cf158921954ce9310214359f76e3a3ef96d93"} Sep 08 18:39:10.672693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.672660 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc"] Sep 08 18:39:10.675029 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:39:10.674998 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3594cff_d79f_4fec_97eb_baee86351196.slice/crio-39e7df6d7e323d0bd9f0ee721977678b12c2a0aeb5750eafaf33703b60ecfca7 WatchSource:0}: Error finding container 39e7df6d7e323d0bd9f0ee721977678b12c2a0aeb5750eafaf33703b60ecfca7: Status 404 returned error can't find the container with id 39e7df6d7e323d0bd9f0ee721977678b12c2a0aeb5750eafaf33703b60ecfca7 Sep 08 18:39:10.794052 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.793975 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-596b5cb86-thw5s"] Sep 08 18:39:10.797180 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.797150 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.799791 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.799728 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 08 18:39:10.799915 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.799836 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 08 18:39:10.800002 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.799981 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vd5bl\"" Sep 08 18:39:10.800108 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.800066 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 18:39:10.800108 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.800106 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 18:39:10.800844 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.800823 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 18:39:10.800987 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.800882 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 18:39:10.800987 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.800823 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 18:39:10.806263 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.805920 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 08 18:39:10.806590 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.806535 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-596b5cb86-thw5s"] Sep 08 18:39:10.885822 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.885787 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-oauth-config\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.885996 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.885834 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx27r\" (UniqueName: \"kubernetes.io/projected/53e31048-2b8e-4b6c-ae05-97d07f30773c-kube-api-access-fx27r\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.885996 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.885887 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-config\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.885996 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.885920 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-serving-cert\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.886153 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.886003 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-service-ca\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.886153 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.886035 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-oauth-serving-cert\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.886153 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.886066 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-trusted-ca-bundle\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.987112 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.987072 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-oauth-config\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.987387 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.987342 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx27r\" (UniqueName: \"kubernetes.io/projected/53e31048-2b8e-4b6c-ae05-97d07f30773c-kube-api-access-fx27r\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.987483 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.987430 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-config\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.987483 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.987471 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-serving-cert\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.987595 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.987533 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-service-ca\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.987595 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.987565 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-oauth-serving-cert\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.987695 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.987597 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-trusted-ca-bundle\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.988869 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.988539 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-service-ca\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.988869 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.988564 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-config\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.989033 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.988888 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-oauth-serving-cert\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.989091 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.989073 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-trusted-ca-bundle\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.991250 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.991219 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-serving-cert\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.991919 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.991878 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-oauth-config\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:10.999696 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:10.999653 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx27r\" (UniqueName: \"kubernetes.io/projected/53e31048-2b8e-4b6c-ae05-97d07f30773c-kube-api-access-fx27r\") pod \"console-596b5cb86-thw5s\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:11.109738 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:11.109535 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:11.641351 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:11.641309 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" event={"ID":"f3594cff-d79f-4fec-97eb-baee86351196","Type":"ContainerStarted","Data":"39e7df6d7e323d0bd9f0ee721977678b12c2a0aeb5750eafaf33703b60ecfca7"} Sep 08 18:39:12.016894 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:12.016809 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:39:12.418142 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:12.418111 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-596b5cb86-thw5s"] Sep 08 18:39:12.647068 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:12.647032 2553 generic.go:356] "Generic (PLEG): container finished" podID="c8ede2fd-b04a-4bfe-be85-b74a047eaf5d" containerID="bf9fe7ade9c00224607bfda3761fa7052cd2c5e1782d0d7569b5e0f443a86193" exitCode=0 Sep 08 18:39:12.647548 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:12.647111 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2htzd" event={"ID":"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d","Type":"ContainerDied","Data":"bf9fe7ade9c00224607bfda3761fa7052cd2c5e1782d0d7569b5e0f443a86193"} Sep 08 18:39:12.649449 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:12.649421 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" event={"ID":"f3594cff-d79f-4fec-97eb-baee86351196","Type":"ContainerStarted","Data":"6c72f835983fe5b1deb858cd82589eecc5083d3952efb69dc03a9901f710256d"} Sep 08 18:39:12.649584 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:12.649483 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" event={"ID":"f3594cff-d79f-4fec-97eb-baee86351196","Type":"ContainerStarted","Data":"8ac8bcf9fe9e1e8ec158f5c02c82ac2f77f435958bfd71cf74c722f66a4fb758"} Sep 08 18:39:12.649584 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:12.649512 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" event={"ID":"f3594cff-d79f-4fec-97eb-baee86351196","Type":"ContainerStarted","Data":"4d67fb7e3582abaf102ddc6fb96c6b4175f655757cf0371c6dac843939a31344"} Sep 08 18:39:12.650841 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:12.650812 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596b5cb86-thw5s" event={"ID":"53e31048-2b8e-4b6c-ae05-97d07f30773c","Type":"ContainerStarted","Data":"e798a977fdc71116d40971b51a776923054ee270d9d93f3f40447e08a4939bb5"} Sep 08 18:39:12.652869 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:12.652847 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pjc2h" event={"ID":"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c","Type":"ContainerStarted","Data":"a106133b08043772730610d054ac8b5852a4fcb1f2f592c3cc45bbb7239bd527"} Sep 08 18:39:12.652978 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:12.652890 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pjc2h" event={"ID":"71e540dd-1e1b-46ab-b6d5-01a0ccf6463c","Type":"ContainerStarted","Data":"79583cc5716c26cbb6fcac0fb9311faa48c3921dd6f3d94d32dc543ffef238d1"} Sep 08 18:39:12.652978 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:12.652932 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pjc2h" Sep 08 18:39:12.683040 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:12.682989 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gd5qc" podStartSLOduration=2.214360707 podStartE2EDuration="3.682971611s" totalImagesPullingTime="1.468610904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:12.681451372 +0000 UTC m=+288.387339605" watchObservedRunningTime="2026-09-08 18:39:12.682971611 +0000 UTC m=+288.388859844" Sep 08 18:39:12.699177 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:12.699138 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pjc2h" podStartSLOduration=251.460308908 podStartE2EDuration="4m13.699124247s" totalImagesPullingTime="2.238815339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:12.698627183 +0000 UTC m=+288.404515415" watchObservedRunningTime="2026-09-08 18:39:12.699124247 +0000 UTC m=+288.405012479" Sep 08 18:39:13.658937 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.658900 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2htzd" event={"ID":"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d","Type":"ContainerStarted","Data":"aa641eb76b3248fe6516ee4c1a56914e99955186697356221ea63323f014fe37"} Sep 08 18:39:13.659515 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.658954 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2htzd" event={"ID":"c8ede2fd-b04a-4bfe-be85-b74a047eaf5d","Type":"ContainerStarted","Data":"b4bf1577595b53ff178eadb77d681352a0faffc9a99d00be3bf4e961ff966911"} Sep 08 18:39:13.676992 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.676947 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2htzd" podStartSLOduration=2.860352915 podStartE2EDuration="4.676934496s" totalImagesPullingTime="1.725576216s" totalInitContainerRuntime="91.005365ms" isStatefulPod=true podCreationTimestamp="2026-09-08 18:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:13.675676014 +0000 UTC m=+289.381564247" watchObservedRunningTime="2026-09-08 18:39:13.676934496 +0000 UTC m=+289.382822728" Sep 08 18:39:13.800513 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.800477 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-54c88f84f-lpg5p"] Sep 08 18:39:13.803045 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.803025 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:13.805551 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.805513 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 08 18:39:13.805675 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.805664 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 08 18:39:13.806011 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.805984 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 08 18:39:13.806292 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.806269 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kmzt6\"" Sep 08 18:39:13.806292 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.806288 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3sdtcam5r31q6\"" Sep 08 18:39:13.806476 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.806264 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 08 18:39:13.812910 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.812890 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54c88f84f-lpg5p"] Sep 08 18:39:13.916873 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.916788 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-secret-metrics-server-client-certs\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:13.917028 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.916884 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-secret-metrics-server-tls\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:13.917028 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.916903 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8kmn\" (UniqueName: \"kubernetes.io/projected/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-kube-api-access-g8kmn\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:13.917028 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.916928 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:13.917028 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.916954 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-audit-log\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:13.917217 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.917057 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-metrics-server-audit-profiles\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:13.917217 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:13.917080 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-client-ca-bundle\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.017697 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.017663 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.017697 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.017698 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-audit-log\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.017937 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.017736 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-metrics-server-audit-profiles\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.017937 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.017795 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-client-ca-bundle\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.017937 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.017862 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-secret-metrics-server-client-certs\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.018088 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.017963 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-secret-metrics-server-tls\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.018088 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.017989 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8kmn\" (UniqueName: \"kubernetes.io/projected/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-kube-api-access-g8kmn\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.018237 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.018205 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-audit-log\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.018515 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.018487 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.018744 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.018726 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-metrics-server-audit-profiles\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.020290 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.020270 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-secret-metrics-server-tls\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.020384 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.020318 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-client-ca-bundle\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.020511 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.020490 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-secret-metrics-server-client-certs\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.025612 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.025586 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8kmn\" (UniqueName: \"kubernetes.io/projected/c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3-kube-api-access-g8kmn\") pod \"metrics-server-54c88f84f-lpg5p\" (UID: \"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3\") " pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.114733 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.114703 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:14.393709 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.393678 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54c88f84f-lpg5p"] Sep 08 18:39:14.398175 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:39:14.398150 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3baf95c_aa39_4ecd_8b61_c9a9b4c90bc3.slice/crio-57731414fc4297bf9af025a0464302be464bcde9afbfa3822231c77318932baa WatchSource:0}: Error finding container 57731414fc4297bf9af025a0464302be464bcde9afbfa3822231c77318932baa: Status 404 returned error can't find the container with id 57731414fc4297bf9af025a0464302be464bcde9afbfa3822231c77318932baa Sep 08 18:39:14.578810 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.578742 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55c57bf558-n9hfq" Sep 08 18:39:14.663339 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:14.663305 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" event={"ID":"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3","Type":"ContainerStarted","Data":"57731414fc4297bf9af025a0464302be464bcde9afbfa3822231c77318932baa"} Sep 08 18:39:16.412951 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:16.412913 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-596b5cb86-thw5s"] Sep 08 18:39:17.030353 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.030284 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-976857789-j5gl2" podUID="fd9c567c-3d00-41b2-862a-66c9b85c6fca" containerName="registry" containerID="cri-o://a06b943e8d97428462b2b452f4176ff65d2dfdae803e569f8b9762795174cc43" gracePeriod=30 Sep 08 18:39:17.271130 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.271107 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:39:17.447155 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.447077 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-bound-sa-token\") pod \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " Sep 08 18:39:17.447566 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.447155 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd9c567c-3d00-41b2-862a-66c9b85c6fca-trusted-ca\") pod \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " Sep 08 18:39:17.447566 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.447196 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd9c567c-3d00-41b2-862a-66c9b85c6fca-ca-trust-extracted\") pod \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " Sep 08 18:39:17.447566 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.447239 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-certificates\") pod \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " Sep 08 18:39:17.447566 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.447271 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd9c567c-3d00-41b2-862a-66c9b85c6fca-image-registry-private-configuration\") pod \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " Sep 08 18:39:17.447566 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.447301 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls\") pod \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " Sep 08 18:39:17.447566 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.447329 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mf85\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-kube-api-access-5mf85\") pod \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " Sep 08 18:39:17.447566 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.447355 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd9c567c-3d00-41b2-862a-66c9b85c6fca-installation-pull-secrets\") pod \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\" (UID: \"fd9c567c-3d00-41b2-862a-66c9b85c6fca\") " Sep 08 18:39:17.447929 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.447652 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9c567c-3d00-41b2-862a-66c9b85c6fca-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fd9c567c-3d00-41b2-862a-66c9b85c6fca" (UID: "fd9c567c-3d00-41b2-862a-66c9b85c6fca"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:39:17.447929 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.447674 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fd9c567c-3d00-41b2-862a-66c9b85c6fca" (UID: "fd9c567c-3d00-41b2-862a-66c9b85c6fca"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:39:17.449810 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.449745 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd9c567c-3d00-41b2-862a-66c9b85c6fca-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fd9c567c-3d00-41b2-862a-66c9b85c6fca" (UID: "fd9c567c-3d00-41b2-862a-66c9b85c6fca"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:39:17.449933 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.449862 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd9c567c-3d00-41b2-862a-66c9b85c6fca-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fd9c567c-3d00-41b2-862a-66c9b85c6fca" (UID: "fd9c567c-3d00-41b2-862a-66c9b85c6fca"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:39:17.449933 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.449910 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fd9c567c-3d00-41b2-862a-66c9b85c6fca" (UID: "fd9c567c-3d00-41b2-862a-66c9b85c6fca"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:39:17.450008 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.449974 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fd9c567c-3d00-41b2-862a-66c9b85c6fca" (UID: "fd9c567c-3d00-41b2-862a-66c9b85c6fca"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:39:17.450085 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.450067 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-kube-api-access-5mf85" (OuterVolumeSpecName: "kube-api-access-5mf85") pod "fd9c567c-3d00-41b2-862a-66c9b85c6fca" (UID: "fd9c567c-3d00-41b2-862a-66c9b85c6fca"). InnerVolumeSpecName "kube-api-access-5mf85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:39:17.456595 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.456570 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9c567c-3d00-41b2-862a-66c9b85c6fca-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fd9c567c-3d00-41b2-862a-66c9b85c6fca" (UID: "fd9c567c-3d00-41b2-862a-66c9b85c6fca"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:39:17.548588 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.548552 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-tls\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:39:17.548588 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.548583 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mf85\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-kube-api-access-5mf85\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:39:17.548588 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.548597 2553 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd9c567c-3d00-41b2-862a-66c9b85c6fca-installation-pull-secrets\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:39:17.548813 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.548609 2553 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd9c567c-3d00-41b2-862a-66c9b85c6fca-bound-sa-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:39:17.548813 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.548620 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd9c567c-3d00-41b2-862a-66c9b85c6fca-trusted-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:39:17.548813 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.548631 2553 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd9c567c-3d00-41b2-862a-66c9b85c6fca-ca-trust-extracted\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:39:17.548813 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.548641 2553 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd9c567c-3d00-41b2-862a-66c9b85c6fca-registry-certificates\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:39:17.548813 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.548653 2553 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd9c567c-3d00-41b2-862a-66c9b85c6fca-image-registry-private-configuration\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:39:17.677669 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.677634 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596b5cb86-thw5s" event={"ID":"53e31048-2b8e-4b6c-ae05-97d07f30773c","Type":"ContainerStarted","Data":"62acd86136aa61ce91e5233c6a8d61812561799ff31533fff89a44ab8340c89f"} Sep 08 18:39:17.678898 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.678871 2553 generic.go:356] "Generic (PLEG): container finished" podID="fd9c567c-3d00-41b2-862a-66c9b85c6fca" containerID="a06b943e8d97428462b2b452f4176ff65d2dfdae803e569f8b9762795174cc43" exitCode=0 Sep 08 18:39:17.679019 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.678909 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-976857789-j5gl2" Sep 08 18:39:17.679019 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.678952 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-976857789-j5gl2" event={"ID":"fd9c567c-3d00-41b2-862a-66c9b85c6fca","Type":"ContainerDied","Data":"a06b943e8d97428462b2b452f4176ff65d2dfdae803e569f8b9762795174cc43"} Sep 08 18:39:17.679019 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.679000 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-976857789-j5gl2" event={"ID":"fd9c567c-3d00-41b2-862a-66c9b85c6fca","Type":"ContainerDied","Data":"29d6f684dca25af313fd95d4d977d55dc4a8a9c510549b6f26dc765325f8decf"} Sep 08 18:39:17.679159 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.679026 2553 scope.go:117] "RemoveContainer" containerID="a06b943e8d97428462b2b452f4176ff65d2dfdae803e569f8b9762795174cc43" Sep 08 18:39:17.680631 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.680608 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" event={"ID":"c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3","Type":"ContainerStarted","Data":"f56e060c8efd1b53015397aa07595c72e9cfaf4ad5709e7072360882f6b4f525"} Sep 08 18:39:17.687601 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.687581 2553 scope.go:117] "RemoveContainer" containerID="a06b943e8d97428462b2b452f4176ff65d2dfdae803e569f8b9762795174cc43" Sep 08 18:39:17.687924 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:39:17.687903 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06b943e8d97428462b2b452f4176ff65d2dfdae803e569f8b9762795174cc43\": container with ID starting with a06b943e8d97428462b2b452f4176ff65d2dfdae803e569f8b9762795174cc43 not found: ID does not exist" containerID="a06b943e8d97428462b2b452f4176ff65d2dfdae803e569f8b9762795174cc43" Sep 08 18:39:17.687999 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.687936 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06b943e8d97428462b2b452f4176ff65d2dfdae803e569f8b9762795174cc43"} err="failed to get container status \"a06b943e8d97428462b2b452f4176ff65d2dfdae803e569f8b9762795174cc43\": rpc error: code = NotFound desc = could not find container \"a06b943e8d97428462b2b452f4176ff65d2dfdae803e569f8b9762795174cc43\": container with ID starting with a06b943e8d97428462b2b452f4176ff65d2dfdae803e569f8b9762795174cc43 not found: ID does not exist" Sep 08 18:39:17.694466 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.694429 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-596b5cb86-thw5s" podStartSLOduration=3.174638836 podStartE2EDuration="7.694415711s" totalImagesPullingTime="4.519776875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:17.693988533 +0000 UTC m=+293.399876775" watchObservedRunningTime="2026-09-08 18:39:17.694415711 +0000 UTC m=+293.400303942" Sep 08 18:39:17.710585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.710515 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" podStartSLOduration=2.167177844 podStartE2EDuration="4.710501844s" totalImagesPullingTime="2.543324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:39:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:39:17.708904129 +0000 UTC m=+293.414792360" watchObservedRunningTime="2026-09-08 18:39:17.710501844 +0000 UTC m=+293.416390075" Sep 08 18:39:17.727762 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.727735 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-976857789-j5gl2"] Sep 08 18:39:17.738694 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:17.738671 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-976857789-j5gl2"] Sep 08 18:39:18.744527 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:18.744493 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd9c567c-3d00-41b2-862a-66c9b85c6fca" path="/var/lib/kubelet/pods/fd9c567c-3d00-41b2-862a-66c9b85c6fca/volumes" Sep 08 18:39:19.739877 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:39:19.739817 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-sjrbw" podUID="40b6e42c-6d96-445d-8b83-108320730adb" Sep 08 18:39:21.110524 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:21.110488 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:22.661297 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:22.661265 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pjc2h" Sep 08 18:39:24.623341 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:24.623304 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 18:39:24.624060 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:24.624036 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 18:39:24.627921 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:24.627905 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 18:39:24.628514 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:24.628492 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 18:39:24.635343 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:24.635324 2553 kubelet.go:1648] "Image garbage collection succeeded" Sep 08 18:39:33.738599 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:33.738564 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:39:34.115006 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:34.114923 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:34.115006 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:34.114981 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:42.697893 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:42.697834 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-596b5cb86-thw5s" podUID="53e31048-2b8e-4b6c-ae05-97d07f30773c" containerName="console" containerID="cri-o://62acd86136aa61ce91e5233c6a8d61812561799ff31533fff89a44ab8340c89f" gracePeriod=15 Sep 08 18:39:42.929286 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:42.929257 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:43.031260 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.031178 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-trusted-ca-bundle\") pod \"53e31048-2b8e-4b6c-ae05-97d07f30773c\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " Sep 08 18:39:43.031260 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.031223 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-oauth-config\") pod \"53e31048-2b8e-4b6c-ae05-97d07f30773c\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " Sep 08 18:39:43.031471 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.031270 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx27r\" (UniqueName: \"kubernetes.io/projected/53e31048-2b8e-4b6c-ae05-97d07f30773c-kube-api-access-fx27r\") pod \"53e31048-2b8e-4b6c-ae05-97d07f30773c\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " Sep 08 18:39:43.031471 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.031396 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-oauth-serving-cert\") pod \"53e31048-2b8e-4b6c-ae05-97d07f30773c\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " Sep 08 18:39:43.031471 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.031438 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-config\") pod \"53e31048-2b8e-4b6c-ae05-97d07f30773c\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " Sep 08 18:39:43.031471 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.031467 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-serving-cert\") pod \"53e31048-2b8e-4b6c-ae05-97d07f30773c\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " Sep 08 18:39:43.031662 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.031501 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-service-ca\") pod \"53e31048-2b8e-4b6c-ae05-97d07f30773c\" (UID: \"53e31048-2b8e-4b6c-ae05-97d07f30773c\") " Sep 08 18:39:43.031807 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.031781 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "53e31048-2b8e-4b6c-ae05-97d07f30773c" (UID: "53e31048-2b8e-4b6c-ae05-97d07f30773c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:39:43.031807 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.031794 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "53e31048-2b8e-4b6c-ae05-97d07f30773c" (UID: "53e31048-2b8e-4b6c-ae05-97d07f30773c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:39:43.031961 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.031809 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-config" (OuterVolumeSpecName: "console-config") pod "53e31048-2b8e-4b6c-ae05-97d07f30773c" (UID: "53e31048-2b8e-4b6c-ae05-97d07f30773c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:39:43.032022 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.031996 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-service-ca" (OuterVolumeSpecName: "service-ca") pod "53e31048-2b8e-4b6c-ae05-97d07f30773c" (UID: "53e31048-2b8e-4b6c-ae05-97d07f30773c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:39:43.033655 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.033620 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "53e31048-2b8e-4b6c-ae05-97d07f30773c" (UID: "53e31048-2b8e-4b6c-ae05-97d07f30773c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:39:43.033775 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.033672 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e31048-2b8e-4b6c-ae05-97d07f30773c-kube-api-access-fx27r" (OuterVolumeSpecName: "kube-api-access-fx27r") pod "53e31048-2b8e-4b6c-ae05-97d07f30773c" (UID: "53e31048-2b8e-4b6c-ae05-97d07f30773c"). InnerVolumeSpecName "kube-api-access-fx27r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:39:43.033873 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.033850 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "53e31048-2b8e-4b6c-ae05-97d07f30773c" (UID: "53e31048-2b8e-4b6c-ae05-97d07f30773c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:39:43.132167 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.132140 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx27r\" (UniqueName: \"kubernetes.io/projected/53e31048-2b8e-4b6c-ae05-97d07f30773c-kube-api-access-fx27r\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:39:43.132167 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.132163 2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-oauth-serving-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:39:43.132167 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.132173 2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:39:43.132352 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.132182 2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-serving-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:39:43.132352 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.132190 2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:39:43.132352 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.132198 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53e31048-2b8e-4b6c-ae05-97d07f30773c-trusted-ca-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:39:43.132352 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.132206 2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53e31048-2b8e-4b6c-ae05-97d07f30773c-console-oauth-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:39:43.777748 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.777709 2553 generic.go:356] "Generic (PLEG): container finished" podID="53e31048-2b8e-4b6c-ae05-97d07f30773c" containerID="62acd86136aa61ce91e5233c6a8d61812561799ff31533fff89a44ab8340c89f" exitCode=0 Sep 08 18:39:43.777748 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.777742 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-596b5cb86-thw5s" Sep 08 18:39:43.778270 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.777809 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596b5cb86-thw5s" event={"ID":"53e31048-2b8e-4b6c-ae05-97d07f30773c","Type":"ContainerDied","Data":"62acd86136aa61ce91e5233c6a8d61812561799ff31533fff89a44ab8340c89f"} Sep 08 18:39:43.778270 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.777873 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-596b5cb86-thw5s" event={"ID":"53e31048-2b8e-4b6c-ae05-97d07f30773c","Type":"ContainerDied","Data":"e798a977fdc71116d40971b51a776923054ee270d9d93f3f40447e08a4939bb5"} Sep 08 18:39:43.778270 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.777906 2553 scope.go:117] "RemoveContainer" containerID="62acd86136aa61ce91e5233c6a8d61812561799ff31533fff89a44ab8340c89f" Sep 08 18:39:43.786658 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.786644 2553 scope.go:117] "RemoveContainer" containerID="62acd86136aa61ce91e5233c6a8d61812561799ff31533fff89a44ab8340c89f" Sep 08 18:39:43.786894 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:39:43.786877 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62acd86136aa61ce91e5233c6a8d61812561799ff31533fff89a44ab8340c89f\": container with ID starting with 62acd86136aa61ce91e5233c6a8d61812561799ff31533fff89a44ab8340c89f not found: ID does not exist" containerID="62acd86136aa61ce91e5233c6a8d61812561799ff31533fff89a44ab8340c89f" Sep 08 18:39:43.786931 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.786902 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62acd86136aa61ce91e5233c6a8d61812561799ff31533fff89a44ab8340c89f"} err="failed to get container status \"62acd86136aa61ce91e5233c6a8d61812561799ff31533fff89a44ab8340c89f\": rpc error: code = NotFound desc = could not find container \"62acd86136aa61ce91e5233c6a8d61812561799ff31533fff89a44ab8340c89f\": container with ID starting with 62acd86136aa61ce91e5233c6a8d61812561799ff31533fff89a44ab8340c89f not found: ID does not exist" Sep 08 18:39:43.797804 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.797783 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-596b5cb86-thw5s"] Sep 08 18:39:43.801031 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:43.801010 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-596b5cb86-thw5s"] Sep 08 18:39:44.743122 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:44.743090 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53e31048-2b8e-4b6c-ae05-97d07f30773c" path="/var/lib/kubelet/pods/53e31048-2b8e-4b6c-ae05-97d07f30773c/volumes" Sep 08 18:39:54.120749 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:54.120717 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:54.124471 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:54.124448 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-54c88f84f-lpg5p" Sep 08 18:39:55.820633 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:55.820597 2553 generic.go:356] "Generic (PLEG): container finished" podID="4fa729ff-ae68-404f-a6c0-a1c87566f68c" containerID="b02ec9e8907dcaef374ad202752d873aa95b20f21ebc2635ed413f45728faf9c" exitCode=0 Sep 08 18:39:55.821058 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:55.820653 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" event={"ID":"4fa729ff-ae68-404f-a6c0-a1c87566f68c","Type":"ContainerDied","Data":"b02ec9e8907dcaef374ad202752d873aa95b20f21ebc2635ed413f45728faf9c"} Sep 08 18:39:55.821058 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:55.820950 2553 scope.go:117] "RemoveContainer" containerID="b02ec9e8907dcaef374ad202752d873aa95b20f21ebc2635ed413f45728faf9c" Sep 08 18:39:56.828483 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:39:56.828445 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-phjbj" event={"ID":"4fa729ff-ae68-404f-a6c0-a1c87566f68c","Type":"ContainerStarted","Data":"fc695b75e1c1e4d99cb7e8f9992e733731fb81a7a865205354283881fcee15f1"} Sep 08 18:40:37.567903 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.567871 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d44b4b4b5-7m77c"] Sep 08 18:40:37.568298 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.568142 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53e31048-2b8e-4b6c-ae05-97d07f30773c" containerName="console" Sep 08 18:40:37.568298 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.568153 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="53e31048-2b8e-4b6c-ae05-97d07f30773c" containerName="console" Sep 08 18:40:37.568298 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.568162 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd9c567c-3d00-41b2-862a-66c9b85c6fca" containerName="registry" Sep 08 18:40:37.568298 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.568168 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd9c567c-3d00-41b2-862a-66c9b85c6fca" containerName="registry" Sep 08 18:40:37.568298 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.568214 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="53e31048-2b8e-4b6c-ae05-97d07f30773c" containerName="console" Sep 08 18:40:37.568298 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.568222 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd9c567c-3d00-41b2-862a-66c9b85c6fca" containerName="registry" Sep 08 18:40:37.570974 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.570948 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.573737 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.573707 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 18:40:37.573872 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.573707 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vd5bl\"" Sep 08 18:40:37.574888 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.574867 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 18:40:37.575054 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.574939 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 18:40:37.575254 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.575231 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 08 18:40:37.575339 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.575245 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 18:40:37.575339 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.575328 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 08 18:40:37.575468 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.575340 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 18:40:37.581185 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.581161 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 08 18:40:37.586872 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.586853 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d44b4b4b5-7m77c"] Sep 08 18:40:37.661035 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.661004 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kzqx\" (UniqueName: \"kubernetes.io/projected/3cef542a-40e1-4eb9-ae19-f5e973fe2598-kube-api-access-8kzqx\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.661035 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.661038 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-trusted-ca-bundle\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.661203 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.661102 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-oauth-serving-cert\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.661203 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.661141 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-config\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.661203 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.661158 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-serving-cert\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.661203 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.661185 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-oauth-config\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.661203 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.661201 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-service-ca\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.762327 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.762292 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-config\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.762327 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.762327 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-serving-cert\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.762519 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.762391 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-oauth-config\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.762519 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.762417 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-service-ca\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.762519 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.762447 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kzqx\" (UniqueName: \"kubernetes.io/projected/3cef542a-40e1-4eb9-ae19-f5e973fe2598-kube-api-access-8kzqx\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.762665 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.762595 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-trusted-ca-bundle\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.762718 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.762684 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-oauth-serving-cert\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.763163 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.763135 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-config\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.763293 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.763204 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-service-ca\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.763354 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.763321 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-oauth-serving-cert\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.763809 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.763787 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-trusted-ca-bundle\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.764882 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.764857 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-oauth-config\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.765021 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.765002 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-serving-cert\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.771499 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.771480 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kzqx\" (UniqueName: \"kubernetes.io/projected/3cef542a-40e1-4eb9-ae19-f5e973fe2598-kube-api-access-8kzqx\") pod \"console-6d44b4b4b5-7m77c\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:37.881765 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:37.881733 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:38.160472 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:38.160446 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d44b4b4b5-7m77c"] Sep 08 18:40:38.162742 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:40:38.162715 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cef542a_40e1_4eb9_ae19_f5e973fe2598.slice/crio-cda607c93ed287739d0e50399a9b9021bccad7dff115a720e68383cbfc038e8e WatchSource:0}: Error finding container cda607c93ed287739d0e50399a9b9021bccad7dff115a720e68383cbfc038e8e: Status 404 returned error can't find the container with id cda607c93ed287739d0e50399a9b9021bccad7dff115a720e68383cbfc038e8e Sep 08 18:40:38.670549 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:38.670512 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:40:38.672772 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:38.672754 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b6e42c-6d96-445d-8b83-108320730adb-metrics-certs\") pod \"network-metrics-daemon-sjrbw\" (UID: \"40b6e42c-6d96-445d-8b83-108320730adb\") " pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:40:38.842231 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:38.842205 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z79bg\"" Sep 08 18:40:38.850173 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:38.850156 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sjrbw" Sep 08 18:40:38.980409 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:38.980307 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d44b4b4b5-7m77c" event={"ID":"3cef542a-40e1-4eb9-ae19-f5e973fe2598","Type":"ContainerStarted","Data":"5e542a262bc621368a68f7f026aba8dc0a42e21f406cef4e9be267275347535a"} Sep 08 18:40:38.980409 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:38.980359 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d44b4b4b5-7m77c" event={"ID":"3cef542a-40e1-4eb9-ae19-f5e973fe2598","Type":"ContainerStarted","Data":"cda607c93ed287739d0e50399a9b9021bccad7dff115a720e68383cbfc038e8e"} Sep 08 18:40:38.996532 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:38.996482 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d44b4b4b5-7m77c" podStartSLOduration=1.9964657319999999 podStartE2EDuration="1.996465732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:40:38.996134366 +0000 UTC m=+374.702022609" watchObservedRunningTime="2026-09-08 18:40:38.996465732 +0000 UTC m=+374.702353969" Sep 08 18:40:39.115342 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:39.115216 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sjrbw"] Sep 08 18:40:39.117625 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:40:39.117597 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b6e42c_6d96_445d_8b83_108320730adb.slice/crio-3fd8bc115d2fffefa064dd32a2065e4d65c331f25af7fb681b5608507056aef5 WatchSource:0}: Error finding container 3fd8bc115d2fffefa064dd32a2065e4d65c331f25af7fb681b5608507056aef5: Status 404 returned error can't find the container with id 3fd8bc115d2fffefa064dd32a2065e4d65c331f25af7fb681b5608507056aef5 Sep 08 18:40:39.119618 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:39.119603 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 18:40:39.984760 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:39.984722 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sjrbw" event={"ID":"40b6e42c-6d96-445d-8b83-108320730adb","Type":"ContainerStarted","Data":"3fd8bc115d2fffefa064dd32a2065e4d65c331f25af7fb681b5608507056aef5"} Sep 08 18:40:40.990219 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:40.990179 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sjrbw" event={"ID":"40b6e42c-6d96-445d-8b83-108320730adb","Type":"ContainerStarted","Data":"89e8604311a7e592764ff578fc2d890eabc84b60a3fade1dbe0f0c1b8ebac914"} Sep 08 18:40:40.990642 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:40.990236 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sjrbw" event={"ID":"40b6e42c-6d96-445d-8b83-108320730adb","Type":"ContainerStarted","Data":"c955c0eb7f99bfa742155594a44d192d95c398e628175dd03f97c391a16a2c0a"} Sep 08 18:40:41.005644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:41.005601 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sjrbw" podStartSLOduration=374.867918053 podStartE2EDuration="6m16.005588767s" totalImagesPullingTime="1.137670714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:40:41.004517416 +0000 UTC m=+376.710405650" watchObservedRunningTime="2026-09-08 18:40:41.005588767 +0000 UTC m=+376.711477000" Sep 08 18:40:47.882567 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:47.882514 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:47.883156 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:47.882591 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:47.887192 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:47.887169 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:40:48.024351 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:40:48.024321 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:41:49.796439 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.796349 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-88f8c8bfb-q9rdw"] Sep 08 18:41:49.798926 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.798894 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.811777 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.811742 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88f8c8bfb-q9rdw"] Sep 08 18:41:49.816394 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.814177 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-trusted-ca-bundle\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.816394 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.814236 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-service-ca\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.816394 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.814268 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh8pk\" (UniqueName: \"kubernetes.io/projected/024643c8-804d-4e3e-9141-1e545cb482dc-kube-api-access-zh8pk\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.816394 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.814323 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/024643c8-804d-4e3e-9141-1e545cb482dc-console-serving-cert\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.816394 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.814392 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/024643c8-804d-4e3e-9141-1e545cb482dc-console-oauth-config\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.816394 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.814422 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-console-config\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.816394 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.814455 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-oauth-serving-cert\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.915331 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.915298 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/024643c8-804d-4e3e-9141-1e545cb482dc-console-oauth-config\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.915331 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.915335 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-console-config\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.915510 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.915381 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-oauth-serving-cert\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.915510 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.915435 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-trusted-ca-bundle\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.915510 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.915467 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-service-ca\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.915510 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.915483 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8pk\" (UniqueName: \"kubernetes.io/projected/024643c8-804d-4e3e-9141-1e545cb482dc-kube-api-access-zh8pk\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.915685 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.915532 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/024643c8-804d-4e3e-9141-1e545cb482dc-console-serving-cert\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.916169 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.916137 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-console-config\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.916336 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.916311 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-trusted-ca-bundle\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.916336 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.916156 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-service-ca\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.916524 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.916156 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-oauth-serving-cert\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.918018 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.917995 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/024643c8-804d-4e3e-9141-1e545cb482dc-console-serving-cert\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.918097 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.918005 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/024643c8-804d-4e3e-9141-1e545cb482dc-console-oauth-config\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:49.922715 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:49.922694 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8pk\" (UniqueName: \"kubernetes.io/projected/024643c8-804d-4e3e-9141-1e545cb482dc-kube-api-access-zh8pk\") pod \"console-88f8c8bfb-q9rdw\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:50.108827 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:50.108744 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:41:50.376997 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:50.376967 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88f8c8bfb-q9rdw"] Sep 08 18:41:50.380874 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:41:50.380850 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024643c8_804d_4e3e_9141_1e545cb482dc.slice/crio-286b40d492e05ad9e1c27250f11b538144c64c8a7c372be70bf06be1a78ca384 WatchSource:0}: Error finding container 286b40d492e05ad9e1c27250f11b538144c64c8a7c372be70bf06be1a78ca384: Status 404 returned error can't find the container with id 286b40d492e05ad9e1c27250f11b538144c64c8a7c372be70bf06be1a78ca384 Sep 08 18:41:51.253180 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:51.253146 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88f8c8bfb-q9rdw" event={"ID":"024643c8-804d-4e3e-9141-1e545cb482dc","Type":"ContainerStarted","Data":"88a8573473936aa96987a9e46a6208735c5fb19b126b5ff372061bacbb026702"} Sep 08 18:41:51.253578 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:51.253194 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88f8c8bfb-q9rdw" event={"ID":"024643c8-804d-4e3e-9141-1e545cb482dc","Type":"ContainerStarted","Data":"286b40d492e05ad9e1c27250f11b538144c64c8a7c372be70bf06be1a78ca384"} Sep 08 18:41:51.269061 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:41:51.269013 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-88f8c8bfb-q9rdw" podStartSLOduration=2.268998557 podStartE2EDuration="2.268998557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:41:51.267578026 +0000 UTC m=+446.973466258" watchObservedRunningTime="2026-09-08 18:41:51.268998557 +0000 UTC m=+446.974886789" Sep 08 18:42:00.109080 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:00.109041 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:42:00.109621 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:00.109109 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:42:00.114079 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:00.114054 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:42:00.292885 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:00.292850 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:42:00.336095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:00.336059 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d44b4b4b5-7m77c"] Sep 08 18:42:25.356676 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.356620 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d44b4b4b5-7m77c" podUID="3cef542a-40e1-4eb9-ae19-f5e973fe2598" containerName="console" containerID="cri-o://5e542a262bc621368a68f7f026aba8dc0a42e21f406cef4e9be267275347535a" gracePeriod=15 Sep 08 18:42:25.587762 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.587739 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:42:25.690316 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.690290 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-serving-cert\") pod \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " Sep 08 18:42:25.690497 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.690321 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-config\") pod \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " Sep 08 18:42:25.690497 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.690481 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-oauth-config\") pod \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " Sep 08 18:42:25.690623 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.690549 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-oauth-serving-cert\") pod \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " Sep 08 18:42:25.690623 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.690584 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-trusted-ca-bundle\") pod \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " Sep 08 18:42:25.690734 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.690649 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kzqx\" (UniqueName: \"kubernetes.io/projected/3cef542a-40e1-4eb9-ae19-f5e973fe2598-kube-api-access-8kzqx\") pod \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " Sep 08 18:42:25.690734 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.690673 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-service-ca\") pod \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\" (UID: \"3cef542a-40e1-4eb9-ae19-f5e973fe2598\") " Sep 08 18:42:25.690734 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.690690 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-config" (OuterVolumeSpecName: "console-config") pod "3cef542a-40e1-4eb9-ae19-f5e973fe2598" (UID: "3cef542a-40e1-4eb9-ae19-f5e973fe2598"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:42:25.690928 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.690900 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3cef542a-40e1-4eb9-ae19-f5e973fe2598" (UID: "3cef542a-40e1-4eb9-ae19-f5e973fe2598"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:42:25.690987 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.690916 2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:42:25.691092 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.691058 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3cef542a-40e1-4eb9-ae19-f5e973fe2598" (UID: "3cef542a-40e1-4eb9-ae19-f5e973fe2598"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:42:25.691191 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.691115 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-service-ca" (OuterVolumeSpecName: "service-ca") pod "3cef542a-40e1-4eb9-ae19-f5e973fe2598" (UID: "3cef542a-40e1-4eb9-ae19-f5e973fe2598"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:42:25.692508 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.692490 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3cef542a-40e1-4eb9-ae19-f5e973fe2598" (UID: "3cef542a-40e1-4eb9-ae19-f5e973fe2598"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:42:25.692981 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.692957 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cef542a-40e1-4eb9-ae19-f5e973fe2598-kube-api-access-8kzqx" (OuterVolumeSpecName: "kube-api-access-8kzqx") pod "3cef542a-40e1-4eb9-ae19-f5e973fe2598" (UID: "3cef542a-40e1-4eb9-ae19-f5e973fe2598"). InnerVolumeSpecName "kube-api-access-8kzqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:42:25.692981 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.692966 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3cef542a-40e1-4eb9-ae19-f5e973fe2598" (UID: "3cef542a-40e1-4eb9-ae19-f5e973fe2598"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:42:25.791660 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.791634 2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-oauth-serving-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:42:25.791660 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.791658 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-trusted-ca-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:42:25.791804 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.791671 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kzqx\" (UniqueName: \"kubernetes.io/projected/3cef542a-40e1-4eb9-ae19-f5e973fe2598-kube-api-access-8kzqx\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:42:25.791804 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.791683 2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3cef542a-40e1-4eb9-ae19-f5e973fe2598-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:42:25.791804 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.791693 2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-serving-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:42:25.791804 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:25.791705 2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3cef542a-40e1-4eb9-ae19-f5e973fe2598-console-oauth-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:42:26.382264 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:26.382226 2553 generic.go:356] "Generic (PLEG): container finished" podID="3cef542a-40e1-4eb9-ae19-f5e973fe2598" containerID="5e542a262bc621368a68f7f026aba8dc0a42e21f406cef4e9be267275347535a" exitCode=0 Sep 08 18:42:26.382695 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:26.382276 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d44b4b4b5-7m77c" Sep 08 18:42:26.382695 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:26.382292 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d44b4b4b5-7m77c" event={"ID":"3cef542a-40e1-4eb9-ae19-f5e973fe2598","Type":"ContainerDied","Data":"5e542a262bc621368a68f7f026aba8dc0a42e21f406cef4e9be267275347535a"} Sep 08 18:42:26.382695 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:26.382342 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d44b4b4b5-7m77c" event={"ID":"3cef542a-40e1-4eb9-ae19-f5e973fe2598","Type":"ContainerDied","Data":"cda607c93ed287739d0e50399a9b9021bccad7dff115a720e68383cbfc038e8e"} Sep 08 18:42:26.382695 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:26.382392 2553 scope.go:117] "RemoveContainer" containerID="5e542a262bc621368a68f7f026aba8dc0a42e21f406cef4e9be267275347535a" Sep 08 18:42:26.390416 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:26.390393 2553 scope.go:117] "RemoveContainer" containerID="5e542a262bc621368a68f7f026aba8dc0a42e21f406cef4e9be267275347535a" Sep 08 18:42:26.390677 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:42:26.390660 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e542a262bc621368a68f7f026aba8dc0a42e21f406cef4e9be267275347535a\": container with ID starting with 5e542a262bc621368a68f7f026aba8dc0a42e21f406cef4e9be267275347535a not found: ID does not exist" containerID="5e542a262bc621368a68f7f026aba8dc0a42e21f406cef4e9be267275347535a" Sep 08 18:42:26.390730 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:26.390692 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e542a262bc621368a68f7f026aba8dc0a42e21f406cef4e9be267275347535a"} err="failed to get container status \"5e542a262bc621368a68f7f026aba8dc0a42e21f406cef4e9be267275347535a\": rpc error: code = NotFound desc = could not find container \"5e542a262bc621368a68f7f026aba8dc0a42e21f406cef4e9be267275347535a\": container with ID starting with 5e542a262bc621368a68f7f026aba8dc0a42e21f406cef4e9be267275347535a not found: ID does not exist" Sep 08 18:42:26.404746 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:26.404714 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d44b4b4b5-7m77c"] Sep 08 18:42:26.408051 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:26.408031 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d44b4b4b5-7m77c"] Sep 08 18:42:26.743690 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:42:26.743613 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cef542a-40e1-4eb9-ae19-f5e973fe2598" path="/var/lib/kubelet/pods/3cef542a-40e1-4eb9-ae19-f5e973fe2598/volumes" Sep 08 18:43:07.364450 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.364413 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc"] Sep 08 18:43:07.364921 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.364687 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cef542a-40e1-4eb9-ae19-f5e973fe2598" containerName="console" Sep 08 18:43:07.364921 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.364698 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cef542a-40e1-4eb9-ae19-f5e973fe2598" containerName="console" Sep 08 18:43:07.364921 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.364756 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cef542a-40e1-4eb9-ae19-f5e973fe2598" containerName="console" Sep 08 18:43:07.367769 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.367754 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" Sep 08 18:43:07.371944 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.371925 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 18:43:07.373150 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.373134 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f66nj\"" Sep 08 18:43:07.373250 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.373152 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 18:43:07.378588 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.378565 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc"] Sep 08 18:43:07.396563 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.396542 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0675dc3-1859-4008-8a7f-eff618ff39fa-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc\" (UID: \"f0675dc3-1859-4008-8a7f-eff618ff39fa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" Sep 08 18:43:07.396684 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.396578 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j27bq\" (UniqueName: \"kubernetes.io/projected/f0675dc3-1859-4008-8a7f-eff618ff39fa-kube-api-access-j27bq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc\" (UID: \"f0675dc3-1859-4008-8a7f-eff618ff39fa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" Sep 08 18:43:07.396684 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.396622 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0675dc3-1859-4008-8a7f-eff618ff39fa-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc\" (UID: \"f0675dc3-1859-4008-8a7f-eff618ff39fa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" Sep 08 18:43:07.497813 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.497784 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j27bq\" (UniqueName: \"kubernetes.io/projected/f0675dc3-1859-4008-8a7f-eff618ff39fa-kube-api-access-j27bq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc\" (UID: \"f0675dc3-1859-4008-8a7f-eff618ff39fa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" Sep 08 18:43:07.497918 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.497853 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0675dc3-1859-4008-8a7f-eff618ff39fa-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc\" (UID: \"f0675dc3-1859-4008-8a7f-eff618ff39fa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" Sep 08 18:43:07.497918 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.497893 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0675dc3-1859-4008-8a7f-eff618ff39fa-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc\" (UID: \"f0675dc3-1859-4008-8a7f-eff618ff39fa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" Sep 08 18:43:07.498209 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.498191 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0675dc3-1859-4008-8a7f-eff618ff39fa-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc\" (UID: \"f0675dc3-1859-4008-8a7f-eff618ff39fa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" Sep 08 18:43:07.498246 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.498226 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0675dc3-1859-4008-8a7f-eff618ff39fa-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc\" (UID: \"f0675dc3-1859-4008-8a7f-eff618ff39fa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" Sep 08 18:43:07.507802 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.507781 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j27bq\" (UniqueName: \"kubernetes.io/projected/f0675dc3-1859-4008-8a7f-eff618ff39fa-kube-api-access-j27bq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc\" (UID: \"f0675dc3-1859-4008-8a7f-eff618ff39fa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" Sep 08 18:43:07.676611 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.676580 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" Sep 08 18:43:07.956954 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:07.956932 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc"] Sep 08 18:43:07.959314 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:43:07.959277 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0675dc3_1859_4008_8a7f_eff618ff39fa.slice/crio-ecf511de502fe65faa08553f428699dd2ebef3fd27b8903fc4842ede66fd330d WatchSource:0}: Error finding container ecf511de502fe65faa08553f428699dd2ebef3fd27b8903fc4842ede66fd330d: Status 404 returned error can't find the container with id ecf511de502fe65faa08553f428699dd2ebef3fd27b8903fc4842ede66fd330d Sep 08 18:43:08.535550 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:08.535515 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" event={"ID":"f0675dc3-1859-4008-8a7f-eff618ff39fa","Type":"ContainerStarted","Data":"ecf511de502fe65faa08553f428699dd2ebef3fd27b8903fc4842ede66fd330d"} Sep 08 18:43:15.566837 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:15.566803 2553 generic.go:356] "Generic (PLEG): container finished" podID="f0675dc3-1859-4008-8a7f-eff618ff39fa" containerID="f3410214c0c6bb812105d54f3b550a23da3192cb1592e4181d99eb08026d04e8" exitCode=0 Sep 08 18:43:15.567333 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:15.566866 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" event={"ID":"f0675dc3-1859-4008-8a7f-eff618ff39fa","Type":"ContainerDied","Data":"f3410214c0c6bb812105d54f3b550a23da3192cb1592e4181d99eb08026d04e8"} Sep 08 18:43:25.607507 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:25.607469 2553 generic.go:356] "Generic (PLEG): container finished" podID="f0675dc3-1859-4008-8a7f-eff618ff39fa" containerID="c09ad084fdc70b0991614fb19bd7de694b95b537e9b3a4f6571b6237e552a0de" exitCode=0 Sep 08 18:43:25.607875 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:25.607550 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" event={"ID":"f0675dc3-1859-4008-8a7f-eff618ff39fa","Type":"ContainerDied","Data":"c09ad084fdc70b0991614fb19bd7de694b95b537e9b3a4f6571b6237e552a0de"} Sep 08 18:43:32.635783 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:32.635743 2553 generic.go:356] "Generic (PLEG): container finished" podID="f0675dc3-1859-4008-8a7f-eff618ff39fa" containerID="4ec2eebc305109cacaabd5f02d6143a9d6ff1e2039d611207f7cccb01b58d117" exitCode=0 Sep 08 18:43:32.636123 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:32.635817 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" event={"ID":"f0675dc3-1859-4008-8a7f-eff618ff39fa","Type":"ContainerDied","Data":"4ec2eebc305109cacaabd5f02d6143a9d6ff1e2039d611207f7cccb01b58d117"} Sep 08 18:43:33.756750 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:33.756725 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" Sep 08 18:43:33.917420 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:33.917309 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j27bq\" (UniqueName: \"kubernetes.io/projected/f0675dc3-1859-4008-8a7f-eff618ff39fa-kube-api-access-j27bq\") pod \"f0675dc3-1859-4008-8a7f-eff618ff39fa\" (UID: \"f0675dc3-1859-4008-8a7f-eff618ff39fa\") " Sep 08 18:43:33.917420 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:33.917349 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0675dc3-1859-4008-8a7f-eff618ff39fa-bundle\") pod \"f0675dc3-1859-4008-8a7f-eff618ff39fa\" (UID: \"f0675dc3-1859-4008-8a7f-eff618ff39fa\") " Sep 08 18:43:33.917632 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:33.917424 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0675dc3-1859-4008-8a7f-eff618ff39fa-util\") pod \"f0675dc3-1859-4008-8a7f-eff618ff39fa\" (UID: \"f0675dc3-1859-4008-8a7f-eff618ff39fa\") " Sep 08 18:43:33.918201 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:33.918170 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0675dc3-1859-4008-8a7f-eff618ff39fa-bundle" (OuterVolumeSpecName: "bundle") pod "f0675dc3-1859-4008-8a7f-eff618ff39fa" (UID: "f0675dc3-1859-4008-8a7f-eff618ff39fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:43:33.919520 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:33.919496 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0675dc3-1859-4008-8a7f-eff618ff39fa-kube-api-access-j27bq" (OuterVolumeSpecName: "kube-api-access-j27bq") pod "f0675dc3-1859-4008-8a7f-eff618ff39fa" (UID: "f0675dc3-1859-4008-8a7f-eff618ff39fa"). InnerVolumeSpecName "kube-api-access-j27bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:43:33.924224 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:33.924195 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0675dc3-1859-4008-8a7f-eff618ff39fa-util" (OuterVolumeSpecName: "util") pod "f0675dc3-1859-4008-8a7f-eff618ff39fa" (UID: "f0675dc3-1859-4008-8a7f-eff618ff39fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:43:34.018416 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:34.018387 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0675dc3-1859-4008-8a7f-eff618ff39fa-util\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:43:34.018416 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:34.018411 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j27bq\" (UniqueName: \"kubernetes.io/projected/f0675dc3-1859-4008-8a7f-eff618ff39fa-kube-api-access-j27bq\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:43:34.018416 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:34.018422 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0675dc3-1859-4008-8a7f-eff618ff39fa-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:43:34.645654 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:34.645617 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" event={"ID":"f0675dc3-1859-4008-8a7f-eff618ff39fa","Type":"ContainerDied","Data":"ecf511de502fe65faa08553f428699dd2ebef3fd27b8903fc4842ede66fd330d"} Sep 08 18:43:34.645654 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:34.645636 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540qrfjc" Sep 08 18:43:34.645853 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:34.645670 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecf511de502fe65faa08553f428699dd2ebef3fd27b8903fc4842ede66fd330d" Sep 08 18:43:45.490576 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.490544 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-tlpzg"] Sep 08 18:43:45.490948 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.490813 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0675dc3-1859-4008-8a7f-eff618ff39fa" containerName="extract" Sep 08 18:43:45.490948 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.490823 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0675dc3-1859-4008-8a7f-eff618ff39fa" containerName="extract" Sep 08 18:43:45.490948 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.490836 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0675dc3-1859-4008-8a7f-eff618ff39fa" containerName="util" Sep 08 18:43:45.490948 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.490841 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0675dc3-1859-4008-8a7f-eff618ff39fa" containerName="util" Sep 08 18:43:45.490948 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.490853 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0675dc3-1859-4008-8a7f-eff618ff39fa" containerName="pull" Sep 08 18:43:45.490948 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.490858 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0675dc3-1859-4008-8a7f-eff618ff39fa" containerName="pull" Sep 08 18:43:45.490948 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.490911 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0675dc3-1859-4008-8a7f-eff618ff39fa" containerName="extract" Sep 08 18:43:45.492849 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.492831 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-tlpzg" Sep 08 18:43:45.495617 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.495593 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jhgpw\"" Sep 08 18:43:45.496609 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.496592 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 08 18:43:45.496702 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.496591 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 08 18:43:45.506576 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.506556 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-tlpzg"] Sep 08 18:43:45.602371 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.602333 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg"] Sep 08 18:43:45.604188 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.604166 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7fw4\" (UniqueName: \"kubernetes.io/projected/ac47f8e1-5479-48f5-813f-49b0d90b8012-kube-api-access-k7fw4\") pod \"cert-manager-webhook-6486ffd997-tlpzg\" (UID: \"ac47f8e1-5479-48f5-813f-49b0d90b8012\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tlpzg" Sep 08 18:43:45.604251 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.604213 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac47f8e1-5479-48f5-813f-49b0d90b8012-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-tlpzg\" (UID: \"ac47f8e1-5479-48f5-813f-49b0d90b8012\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tlpzg" Sep 08 18:43:45.605047 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.605033 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" Sep 08 18:43:45.608097 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.608069 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 18:43:45.608097 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.608083 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 18:43:45.608610 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.608578 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f66nj\"" Sep 08 18:43:45.616762 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.616742 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg"] Sep 08 18:43:45.704719 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.704695 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7fw4\" (UniqueName: \"kubernetes.io/projected/ac47f8e1-5479-48f5-813f-49b0d90b8012-kube-api-access-k7fw4\") pod \"cert-manager-webhook-6486ffd997-tlpzg\" (UID: \"ac47f8e1-5479-48f5-813f-49b0d90b8012\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tlpzg" Sep 08 18:43:45.704858 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.704726 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pkd2\" (UniqueName: \"kubernetes.io/projected/f63358ce-a1b4-4c19-ad55-44e27b0df268-kube-api-access-9pkd2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg\" (UID: \"f63358ce-a1b4-4c19-ad55-44e27b0df268\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" Sep 08 18:43:45.704858 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.704760 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac47f8e1-5479-48f5-813f-49b0d90b8012-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-tlpzg\" (UID: \"ac47f8e1-5479-48f5-813f-49b0d90b8012\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tlpzg" Sep 08 18:43:45.704858 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.704793 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f63358ce-a1b4-4c19-ad55-44e27b0df268-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg\" (UID: \"f63358ce-a1b4-4c19-ad55-44e27b0df268\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" Sep 08 18:43:45.704858 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.704808 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f63358ce-a1b4-4c19-ad55-44e27b0df268-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg\" (UID: \"f63358ce-a1b4-4c19-ad55-44e27b0df268\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" Sep 08 18:43:45.714334 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.714311 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac47f8e1-5479-48f5-813f-49b0d90b8012-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-tlpzg\" (UID: \"ac47f8e1-5479-48f5-813f-49b0d90b8012\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tlpzg" Sep 08 18:43:45.715013 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.714986 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7fw4\" (UniqueName: \"kubernetes.io/projected/ac47f8e1-5479-48f5-813f-49b0d90b8012-kube-api-access-k7fw4\") pod \"cert-manager-webhook-6486ffd997-tlpzg\" (UID: \"ac47f8e1-5479-48f5-813f-49b0d90b8012\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tlpzg" Sep 08 18:43:45.801641 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.801580 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-tlpzg" Sep 08 18:43:45.805684 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.805661 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f63358ce-a1b4-4c19-ad55-44e27b0df268-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg\" (UID: \"f63358ce-a1b4-4c19-ad55-44e27b0df268\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" Sep 08 18:43:45.805749 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.805692 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f63358ce-a1b4-4c19-ad55-44e27b0df268-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg\" (UID: \"f63358ce-a1b4-4c19-ad55-44e27b0df268\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" Sep 08 18:43:45.805749 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.805723 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pkd2\" (UniqueName: \"kubernetes.io/projected/f63358ce-a1b4-4c19-ad55-44e27b0df268-kube-api-access-9pkd2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg\" (UID: \"f63358ce-a1b4-4c19-ad55-44e27b0df268\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" Sep 08 18:43:45.806031 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.806013 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f63358ce-a1b4-4c19-ad55-44e27b0df268-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg\" (UID: \"f63358ce-a1b4-4c19-ad55-44e27b0df268\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" Sep 08 18:43:45.806099 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.806077 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f63358ce-a1b4-4c19-ad55-44e27b0df268-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg\" (UID: \"f63358ce-a1b4-4c19-ad55-44e27b0df268\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" Sep 08 18:43:45.813861 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.813839 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pkd2\" (UniqueName: \"kubernetes.io/projected/f63358ce-a1b4-4c19-ad55-44e27b0df268-kube-api-access-9pkd2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg\" (UID: \"f63358ce-a1b4-4c19-ad55-44e27b0df268\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" Sep 08 18:43:45.917131 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:45.917101 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" Sep 08 18:43:46.068598 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:46.068569 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-tlpzg"] Sep 08 18:43:46.070886 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:43:46.070859 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac47f8e1_5479_48f5_813f_49b0d90b8012.slice/crio-feb5f932f321af0af79aaedf8f65ed0a5840053b1d22027f5cb995d2fa6cc1b3 WatchSource:0}: Error finding container feb5f932f321af0af79aaedf8f65ed0a5840053b1d22027f5cb995d2fa6cc1b3: Status 404 returned error can't find the container with id feb5f932f321af0af79aaedf8f65ed0a5840053b1d22027f5cb995d2fa6cc1b3 Sep 08 18:43:46.184959 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:46.184933 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg"] Sep 08 18:43:46.186822 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:43:46.186801 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63358ce_a1b4_4c19_ad55_44e27b0df268.slice/crio-697113af65b659e2ec17d43a93099bb22396dfb58c893187ef6c7695a124e3ae WatchSource:0}: Error finding container 697113af65b659e2ec17d43a93099bb22396dfb58c893187ef6c7695a124e3ae: Status 404 returned error can't find the container with id 697113af65b659e2ec17d43a93099bb22396dfb58c893187ef6c7695a124e3ae Sep 08 18:43:46.689798 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:46.689763 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-tlpzg" event={"ID":"ac47f8e1-5479-48f5-813f-49b0d90b8012","Type":"ContainerStarted","Data":"feb5f932f321af0af79aaedf8f65ed0a5840053b1d22027f5cb995d2fa6cc1b3"} Sep 08 18:43:46.691272 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:46.691244 2553 generic.go:356] "Generic (PLEG): container finished" podID="f63358ce-a1b4-4c19-ad55-44e27b0df268" containerID="b535a7efef6370ea2b8a252d6eb22802791c3a9291645a6bdfed750609bf1b39" exitCode=0 Sep 08 18:43:46.691398 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:46.691296 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" event={"ID":"f63358ce-a1b4-4c19-ad55-44e27b0df268","Type":"ContainerDied","Data":"b535a7efef6370ea2b8a252d6eb22802791c3a9291645a6bdfed750609bf1b39"} Sep 08 18:43:46.691398 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:46.691326 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" event={"ID":"f63358ce-a1b4-4c19-ad55-44e27b0df268","Type":"ContainerStarted","Data":"697113af65b659e2ec17d43a93099bb22396dfb58c893187ef6c7695a124e3ae"} Sep 08 18:43:49.709934 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:49.709898 2553 generic.go:356] "Generic (PLEG): container finished" podID="f63358ce-a1b4-4c19-ad55-44e27b0df268" containerID="faf2f28b8676d28c4cd50abecb87698cf3a5444ce45cc2f8ca54b0cb93eac463" exitCode=0 Sep 08 18:43:49.710319 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:49.709986 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" event={"ID":"f63358ce-a1b4-4c19-ad55-44e27b0df268","Type":"ContainerDied","Data":"faf2f28b8676d28c4cd50abecb87698cf3a5444ce45cc2f8ca54b0cb93eac463"} Sep 08 18:43:50.716093 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:50.716060 2553 generic.go:356] "Generic (PLEG): container finished" podID="f63358ce-a1b4-4c19-ad55-44e27b0df268" containerID="37dc2aa45fa15e008ccdc35e109979ffea31c5bfb37ab697efaa7ec3e1b9cfa7" exitCode=0 Sep 08 18:43:50.716492 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:50.716127 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" event={"ID":"f63358ce-a1b4-4c19-ad55-44e27b0df268","Type":"ContainerDied","Data":"37dc2aa45fa15e008ccdc35e109979ffea31c5bfb37ab697efaa7ec3e1b9cfa7"} Sep 08 18:43:51.851825 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:51.851802 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" Sep 08 18:43:51.956087 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:51.956058 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pkd2\" (UniqueName: \"kubernetes.io/projected/f63358ce-a1b4-4c19-ad55-44e27b0df268-kube-api-access-9pkd2\") pod \"f63358ce-a1b4-4c19-ad55-44e27b0df268\" (UID: \"f63358ce-a1b4-4c19-ad55-44e27b0df268\") " Sep 08 18:43:51.956249 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:51.956095 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f63358ce-a1b4-4c19-ad55-44e27b0df268-bundle\") pod \"f63358ce-a1b4-4c19-ad55-44e27b0df268\" (UID: \"f63358ce-a1b4-4c19-ad55-44e27b0df268\") " Sep 08 18:43:51.956249 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:51.956140 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f63358ce-a1b4-4c19-ad55-44e27b0df268-util\") pod \"f63358ce-a1b4-4c19-ad55-44e27b0df268\" (UID: \"f63358ce-a1b4-4c19-ad55-44e27b0df268\") " Sep 08 18:43:51.956567 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:51.956544 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63358ce-a1b4-4c19-ad55-44e27b0df268-bundle" (OuterVolumeSpecName: "bundle") pod "f63358ce-a1b4-4c19-ad55-44e27b0df268" (UID: "f63358ce-a1b4-4c19-ad55-44e27b0df268"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:43:51.958225 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:51.958199 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63358ce-a1b4-4c19-ad55-44e27b0df268-kube-api-access-9pkd2" (OuterVolumeSpecName: "kube-api-access-9pkd2") pod "f63358ce-a1b4-4c19-ad55-44e27b0df268" (UID: "f63358ce-a1b4-4c19-ad55-44e27b0df268"). InnerVolumeSpecName "kube-api-access-9pkd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:43:51.960715 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:51.960683 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63358ce-a1b4-4c19-ad55-44e27b0df268-util" (OuterVolumeSpecName: "util") pod "f63358ce-a1b4-4c19-ad55-44e27b0df268" (UID: "f63358ce-a1b4-4c19-ad55-44e27b0df268"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:43:52.057537 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.057483 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pkd2\" (UniqueName: \"kubernetes.io/projected/f63358ce-a1b4-4c19-ad55-44e27b0df268-kube-api-access-9pkd2\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:43:52.057537 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.057503 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f63358ce-a1b4-4c19-ad55-44e27b0df268-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:43:52.057537 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.057512 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f63358ce-a1b4-4c19-ad55-44e27b0df268-util\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:43:52.725323 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.725286 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" event={"ID":"f63358ce-a1b4-4c19-ad55-44e27b0df268","Type":"ContainerDied","Data":"697113af65b659e2ec17d43a93099bb22396dfb58c893187ef6c7695a124e3ae"} Sep 08 18:43:52.725323 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.725319 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fsvctg" Sep 08 18:43:52.725536 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.725337 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="697113af65b659e2ec17d43a93099bb22396dfb58c893187ef6c7695a124e3ae" Sep 08 18:43:52.845430 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.845400 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-db9zd"] Sep 08 18:43:52.845662 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.845651 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63358ce-a1b4-4c19-ad55-44e27b0df268" containerName="util" Sep 08 18:43:52.845707 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.845663 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63358ce-a1b4-4c19-ad55-44e27b0df268" containerName="util" Sep 08 18:43:52.845707 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.845678 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63358ce-a1b4-4c19-ad55-44e27b0df268" containerName="extract" Sep 08 18:43:52.845707 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.845683 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63358ce-a1b4-4c19-ad55-44e27b0df268" containerName="extract" Sep 08 18:43:52.845707 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.845700 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f63358ce-a1b4-4c19-ad55-44e27b0df268" containerName="pull" Sep 08 18:43:52.845707 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.845705 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f63358ce-a1b4-4c19-ad55-44e27b0df268" containerName="pull" Sep 08 18:43:52.845873 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.845750 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f63358ce-a1b4-4c19-ad55-44e27b0df268" containerName="extract" Sep 08 18:43:52.851410 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.851389 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-db9zd" Sep 08 18:43:52.854346 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.854315 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mzkw7\"" Sep 08 18:43:52.855833 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.855811 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-db9zd"] Sep 08 18:43:52.965304 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.965276 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7t5g\" (UniqueName: \"kubernetes.io/projected/345d3662-b256-4e03-b51a-6e7b71002a42-kube-api-access-p7t5g\") pod \"cert-manager-cainjector-559cdd8c87-db9zd\" (UID: \"345d3662-b256-4e03-b51a-6e7b71002a42\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-db9zd" Sep 08 18:43:52.965469 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:52.965333 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/345d3662-b256-4e03-b51a-6e7b71002a42-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-db9zd\" (UID: \"345d3662-b256-4e03-b51a-6e7b71002a42\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-db9zd" Sep 08 18:43:53.066525 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:53.066449 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/345d3662-b256-4e03-b51a-6e7b71002a42-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-db9zd\" (UID: \"345d3662-b256-4e03-b51a-6e7b71002a42\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-db9zd" Sep 08 18:43:53.066525 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:53.066507 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7t5g\" (UniqueName: \"kubernetes.io/projected/345d3662-b256-4e03-b51a-6e7b71002a42-kube-api-access-p7t5g\") pod \"cert-manager-cainjector-559cdd8c87-db9zd\" (UID: \"345d3662-b256-4e03-b51a-6e7b71002a42\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-db9zd" Sep 08 18:43:53.074418 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:53.074387 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/345d3662-b256-4e03-b51a-6e7b71002a42-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-db9zd\" (UID: \"345d3662-b256-4e03-b51a-6e7b71002a42\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-db9zd" Sep 08 18:43:53.074618 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:53.074599 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7t5g\" (UniqueName: \"kubernetes.io/projected/345d3662-b256-4e03-b51a-6e7b71002a42-kube-api-access-p7t5g\") pod \"cert-manager-cainjector-559cdd8c87-db9zd\" (UID: \"345d3662-b256-4e03-b51a-6e7b71002a42\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-db9zd" Sep 08 18:43:53.160570 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:53.160539 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-db9zd" Sep 08 18:43:53.428206 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:53.428125 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-db9zd"] Sep 08 18:43:53.430698 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:43:53.430671 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod345d3662_b256_4e03_b51a_6e7b71002a42.slice/crio-61f4de0450653ff19f381a585970efba3f8bae44a0cb9ccbc29bf48979937a08 WatchSource:0}: Error finding container 61f4de0450653ff19f381a585970efba3f8bae44a0cb9ccbc29bf48979937a08: Status 404 returned error can't find the container with id 61f4de0450653ff19f381a585970efba3f8bae44a0cb9ccbc29bf48979937a08 Sep 08 18:43:53.730733 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:53.730638 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-db9zd" event={"ID":"345d3662-b256-4e03-b51a-6e7b71002a42","Type":"ContainerStarted","Data":"61f4de0450653ff19f381a585970efba3f8bae44a0cb9ccbc29bf48979937a08"} Sep 08 18:43:56.746212 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:56.746174 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-db9zd" event={"ID":"345d3662-b256-4e03-b51a-6e7b71002a42","Type":"ContainerStarted","Data":"5bead2ae3c668cc60f47fd6b334d7adce5596598ea545735773accc736d3aec0"} Sep 08 18:43:56.747686 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:56.747658 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-tlpzg" event={"ID":"ac47f8e1-5479-48f5-813f-49b0d90b8012","Type":"ContainerStarted","Data":"0954a888f486e0ede07c9ac6a487d1e3c12efc22ae5595c5886d50ba3a2d3db0"} Sep 08 18:43:56.747780 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:56.747711 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-tlpzg" Sep 08 18:43:56.762346 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:56.762302 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-db9zd" podStartSLOduration=1.946589669 podStartE2EDuration="4.762286864s" totalImagesPullingTime="2.815697195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:43:56.760629552 +0000 UTC m=+572.466517787" watchObservedRunningTime="2026-09-08 18:43:56.762286864 +0000 UTC m=+572.468175095" Sep 08 18:43:56.779400 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:43:56.779327 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-tlpzg" podStartSLOduration=1.598021548 podStartE2EDuration="11.779312068s" totalImagesPullingTime="10.18129052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:43:56.778159591 +0000 UTC m=+572.484047822" watchObservedRunningTime="2026-09-08 18:43:56.779312068 +0000 UTC m=+572.485200301" Sep 08 18:44:02.754388 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:02.754343 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-tlpzg" Sep 08 18:44:03.501008 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.500970 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2"] Sep 08 18:44:03.506516 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.506498 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" Sep 08 18:44:03.509023 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.508993 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 18:44:03.509139 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.508993 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 18:44:03.509205 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.509176 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f66nj\"" Sep 08 18:44:03.516184 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.516155 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2"] Sep 08 18:44:03.547162 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.547136 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2\" (UID: \"72e8e7c1-5543-4513-b32d-ad37e4a3cc23\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" Sep 08 18:44:03.547302 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.547178 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9px\" (UniqueName: \"kubernetes.io/projected/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-kube-api-access-tq9px\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2\" (UID: \"72e8e7c1-5543-4513-b32d-ad37e4a3cc23\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" Sep 08 18:44:03.547302 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.547277 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2\" (UID: \"72e8e7c1-5543-4513-b32d-ad37e4a3cc23\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" Sep 08 18:44:03.647841 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.647802 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9px\" (UniqueName: \"kubernetes.io/projected/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-kube-api-access-tq9px\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2\" (UID: \"72e8e7c1-5543-4513-b32d-ad37e4a3cc23\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" Sep 08 18:44:03.648011 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.647882 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2\" (UID: \"72e8e7c1-5543-4513-b32d-ad37e4a3cc23\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" Sep 08 18:44:03.648011 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.647931 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2\" (UID: \"72e8e7c1-5543-4513-b32d-ad37e4a3cc23\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" Sep 08 18:44:03.648255 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.648225 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2\" (UID: \"72e8e7c1-5543-4513-b32d-ad37e4a3cc23\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" Sep 08 18:44:03.648391 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.648266 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2\" (UID: \"72e8e7c1-5543-4513-b32d-ad37e4a3cc23\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" Sep 08 18:44:03.655904 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.655882 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9px\" (UniqueName: \"kubernetes.io/projected/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-kube-api-access-tq9px\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2\" (UID: \"72e8e7c1-5543-4513-b32d-ad37e4a3cc23\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" Sep 08 18:44:03.815546 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:03.815463 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" Sep 08 18:44:04.087236 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:04.087209 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2"] Sep 08 18:44:04.089880 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:44:04.089847 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e8e7c1_5543_4513_b32d_ad37e4a3cc23.slice/crio-8f02a6cfdb822b8cdbb2903e30d9bc2845777634008d2b1cf97d4924269ac473 WatchSource:0}: Error finding container 8f02a6cfdb822b8cdbb2903e30d9bc2845777634008d2b1cf97d4924269ac473: Status 404 returned error can't find the container with id 8f02a6cfdb822b8cdbb2903e30d9bc2845777634008d2b1cf97d4924269ac473 Sep 08 18:44:04.779588 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:04.779554 2553 generic.go:356] "Generic (PLEG): container finished" podID="72e8e7c1-5543-4513-b32d-ad37e4a3cc23" containerID="bf8c5062fa954117e26023f7c24971df8827421d4fb759aee8b71e3706f571b8" exitCode=0 Sep 08 18:44:04.779775 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:04.779635 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" event={"ID":"72e8e7c1-5543-4513-b32d-ad37e4a3cc23","Type":"ContainerDied","Data":"bf8c5062fa954117e26023f7c24971df8827421d4fb759aee8b71e3706f571b8"} Sep 08 18:44:04.779775 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:04.779678 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" event={"ID":"72e8e7c1-5543-4513-b32d-ad37e4a3cc23","Type":"ContainerStarted","Data":"8f02a6cfdb822b8cdbb2903e30d9bc2845777634008d2b1cf97d4924269ac473"} Sep 08 18:44:06.789866 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:06.789832 2553 generic.go:356] "Generic (PLEG): container finished" podID="72e8e7c1-5543-4513-b32d-ad37e4a3cc23" containerID="877854cc8ca5fc3c555f7db573f1dfe0b62a9e42e865dd90f4e5579a5b349545" exitCode=0 Sep 08 18:44:06.790360 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:06.789880 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" event={"ID":"72e8e7c1-5543-4513-b32d-ad37e4a3cc23","Type":"ContainerDied","Data":"877854cc8ca5fc3c555f7db573f1dfe0b62a9e42e865dd90f4e5579a5b349545"} Sep 08 18:44:07.796143 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:07.796105 2553 generic.go:356] "Generic (PLEG): container finished" podID="72e8e7c1-5543-4513-b32d-ad37e4a3cc23" containerID="6c527ff65fd6494673e1aee70ba5b61310eb600ebaf73437dcddbc047f4ad841" exitCode=0 Sep 08 18:44:07.796535 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:07.796187 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" event={"ID":"72e8e7c1-5543-4513-b32d-ad37e4a3cc23","Type":"ContainerDied","Data":"6c527ff65fd6494673e1aee70ba5b61310eb600ebaf73437dcddbc047f4ad841"} Sep 08 18:44:08.915519 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:08.915492 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" Sep 08 18:44:08.992682 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:08.992657 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-util\") pod \"72e8e7c1-5543-4513-b32d-ad37e4a3cc23\" (UID: \"72e8e7c1-5543-4513-b32d-ad37e4a3cc23\") " Sep 08 18:44:08.992801 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:08.992690 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq9px\" (UniqueName: \"kubernetes.io/projected/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-kube-api-access-tq9px\") pod \"72e8e7c1-5543-4513-b32d-ad37e4a3cc23\" (UID: \"72e8e7c1-5543-4513-b32d-ad37e4a3cc23\") " Sep 08 18:44:08.992801 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:08.992766 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-bundle\") pod \"72e8e7c1-5543-4513-b32d-ad37e4a3cc23\" (UID: \"72e8e7c1-5543-4513-b32d-ad37e4a3cc23\") " Sep 08 18:44:08.993563 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:08.993540 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-bundle" (OuterVolumeSpecName: "bundle") pod "72e8e7c1-5543-4513-b32d-ad37e4a3cc23" (UID: "72e8e7c1-5543-4513-b32d-ad37e4a3cc23"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:44:08.994856 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:08.994831 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-kube-api-access-tq9px" (OuterVolumeSpecName: "kube-api-access-tq9px") pod "72e8e7c1-5543-4513-b32d-ad37e4a3cc23" (UID: "72e8e7c1-5543-4513-b32d-ad37e4a3cc23"). InnerVolumeSpecName "kube-api-access-tq9px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:44:08.998912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:08.998888 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-util" (OuterVolumeSpecName: "util") pod "72e8e7c1-5543-4513-b32d-ad37e4a3cc23" (UID: "72e8e7c1-5543-4513-b32d-ad37e4a3cc23"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:44:09.093860 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:09.093796 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-util\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:44:09.093860 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:09.093820 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq9px\" (UniqueName: \"kubernetes.io/projected/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-kube-api-access-tq9px\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:44:09.093860 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:09.093831 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/72e8e7c1-5543-4513-b32d-ad37e4a3cc23-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:44:09.804886 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:09.804862 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" Sep 08 18:44:09.804886 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:09.804868 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afg9kr2" event={"ID":"72e8e7c1-5543-4513-b32d-ad37e4a3cc23","Type":"ContainerDied","Data":"8f02a6cfdb822b8cdbb2903e30d9bc2845777634008d2b1cf97d4924269ac473"} Sep 08 18:44:09.805080 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:09.804906 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f02a6cfdb822b8cdbb2903e30d9bc2845777634008d2b1cf97d4924269ac473" Sep 08 18:44:11.866853 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.866822 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb"] Sep 08 18:44:11.867307 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.867201 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e8e7c1-5543-4513-b32d-ad37e4a3cc23" containerName="extract" Sep 08 18:44:11.867307 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.867218 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e8e7c1-5543-4513-b32d-ad37e4a3cc23" containerName="extract" Sep 08 18:44:11.867307 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.867237 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e8e7c1-5543-4513-b32d-ad37e4a3cc23" containerName="util" Sep 08 18:44:11.867307 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.867245 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e8e7c1-5543-4513-b32d-ad37e4a3cc23" containerName="util" Sep 08 18:44:11.867307 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.867257 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e8e7c1-5543-4513-b32d-ad37e4a3cc23" containerName="pull" Sep 08 18:44:11.867307 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.867266 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e8e7c1-5543-4513-b32d-ad37e4a3cc23" containerName="pull" Sep 08 18:44:11.867633 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.867357 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e8e7c1-5543-4513-b32d-ad37e4a3cc23" containerName="extract" Sep 08 18:44:11.869548 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.869529 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:11.872258 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.872223 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6hblb\"" Sep 08 18:44:11.872417 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.872402 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 08 18:44:11.873750 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.873731 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 08 18:44:11.873854 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.873840 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 08 18:44:11.873913 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.873872 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 08 18:44:11.873971 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.873877 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 08 18:44:11.881732 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.881712 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb"] Sep 08 18:44:11.916195 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.916169 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2-cert\") pod \"lws-controller-manager-68bb56fbd7-kw7fb\" (UID: \"42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:11.916323 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.916202 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2mhc\" (UniqueName: \"kubernetes.io/projected/42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2-kube-api-access-g2mhc\") pod \"lws-controller-manager-68bb56fbd7-kw7fb\" (UID: \"42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:11.916323 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.916233 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2-metrics-cert\") pod \"lws-controller-manager-68bb56fbd7-kw7fb\" (UID: \"42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:11.916461 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:11.916335 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2-manager-config\") pod \"lws-controller-manager-68bb56fbd7-kw7fb\" (UID: \"42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:12.017668 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:12.017633 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2-manager-config\") pod \"lws-controller-manager-68bb56fbd7-kw7fb\" (UID: \"42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:12.017829 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:12.017702 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2-cert\") pod \"lws-controller-manager-68bb56fbd7-kw7fb\" (UID: \"42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:12.017829 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:12.017736 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2mhc\" (UniqueName: \"kubernetes.io/projected/42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2-kube-api-access-g2mhc\") pod \"lws-controller-manager-68bb56fbd7-kw7fb\" (UID: \"42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:12.017829 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:12.017774 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2-metrics-cert\") pod \"lws-controller-manager-68bb56fbd7-kw7fb\" (UID: \"42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:12.018406 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:12.018359 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2-manager-config\") pod \"lws-controller-manager-68bb56fbd7-kw7fb\" (UID: \"42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:12.020190 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:12.020169 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2-cert\") pod \"lws-controller-manager-68bb56fbd7-kw7fb\" (UID: \"42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:12.020283 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:12.020215 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2-metrics-cert\") pod \"lws-controller-manager-68bb56fbd7-kw7fb\" (UID: \"42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:12.025773 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:12.025751 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2mhc\" (UniqueName: \"kubernetes.io/projected/42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2-kube-api-access-g2mhc\") pod \"lws-controller-manager-68bb56fbd7-kw7fb\" (UID: \"42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2\") " pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:12.178992 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:12.178958 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:12.455625 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:12.455601 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb"] Sep 08 18:44:12.457553 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:44:12.457518 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e6a3e1_1a51_4ccb_963a_42c6a0afd8d2.slice/crio-5209b714dc08a17dcf81900844e6a229664fa1e2c0349544ec68e5cc595cf1e4 WatchSource:0}: Error finding container 5209b714dc08a17dcf81900844e6a229664fa1e2c0349544ec68e5cc595cf1e4: Status 404 returned error can't find the container with id 5209b714dc08a17dcf81900844e6a229664fa1e2c0349544ec68e5cc595cf1e4 Sep 08 18:44:12.817763 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:12.817680 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" event={"ID":"42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2","Type":"ContainerStarted","Data":"5209b714dc08a17dcf81900844e6a229664fa1e2c0349544ec68e5cc595cf1e4"} Sep 08 18:44:14.826780 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:14.826742 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" event={"ID":"42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2","Type":"ContainerStarted","Data":"7185140023aa94b76cbc20b535c829595226e3d954d1de3e4052b2d7af59ba37"} Sep 08 18:44:14.827152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:14.826802 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:14.843495 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:14.843443 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" podStartSLOduration=2.190217371 podStartE2EDuration="3.84342544s" totalImagesPullingTime="1.653208069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:44:14.843221577 +0000 UTC m=+590.549109810" watchObservedRunningTime="2026-09-08 18:44:14.84342544 +0000 UTC m=+590.549313673" Sep 08 18:44:19.513453 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.513418 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6"] Sep 08 18:44:19.515837 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.515821 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" Sep 08 18:44:19.518304 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.518285 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 18:44:19.519450 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.519431 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 18:44:19.519450 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.519441 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f66nj\"" Sep 08 18:44:19.524259 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.524240 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6"] Sep 08 18:44:19.583751 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.583717 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppn6m\" (UniqueName: \"kubernetes.io/projected/657214e6-4d00-4d9a-9ed1-7e93621e88e5-kube-api-access-ppn6m\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6\" (UID: \"657214e6-4d00-4d9a-9ed1-7e93621e88e5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" Sep 08 18:44:19.583876 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.583795 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/657214e6-4d00-4d9a-9ed1-7e93621e88e5-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6\" (UID: \"657214e6-4d00-4d9a-9ed1-7e93621e88e5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" Sep 08 18:44:19.583876 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.583813 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/657214e6-4d00-4d9a-9ed1-7e93621e88e5-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6\" (UID: \"657214e6-4d00-4d9a-9ed1-7e93621e88e5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" Sep 08 18:44:19.684587 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.684555 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/657214e6-4d00-4d9a-9ed1-7e93621e88e5-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6\" (UID: \"657214e6-4d00-4d9a-9ed1-7e93621e88e5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" Sep 08 18:44:19.684759 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.684592 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/657214e6-4d00-4d9a-9ed1-7e93621e88e5-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6\" (UID: \"657214e6-4d00-4d9a-9ed1-7e93621e88e5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" Sep 08 18:44:19.684759 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.684636 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppn6m\" (UniqueName: \"kubernetes.io/projected/657214e6-4d00-4d9a-9ed1-7e93621e88e5-kube-api-access-ppn6m\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6\" (UID: \"657214e6-4d00-4d9a-9ed1-7e93621e88e5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" Sep 08 18:44:19.685011 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.684985 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/657214e6-4d00-4d9a-9ed1-7e93621e88e5-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6\" (UID: \"657214e6-4d00-4d9a-9ed1-7e93621e88e5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" Sep 08 18:44:19.685072 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.684999 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/657214e6-4d00-4d9a-9ed1-7e93621e88e5-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6\" (UID: \"657214e6-4d00-4d9a-9ed1-7e93621e88e5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" Sep 08 18:44:19.697491 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.697466 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppn6m\" (UniqueName: \"kubernetes.io/projected/657214e6-4d00-4d9a-9ed1-7e93621e88e5-kube-api-access-ppn6m\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6\" (UID: \"657214e6-4d00-4d9a-9ed1-7e93621e88e5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" Sep 08 18:44:19.825078 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:19.825009 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" Sep 08 18:44:20.103615 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.103590 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6"] Sep 08 18:44:20.104665 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:44:20.104631 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657214e6_4d00_4d9a_9ed1_7e93621e88e5.slice/crio-0383e2e0e6697c8e53a3580a9709fae9bc20c037491324f3abcd11f90cf19c88 WatchSource:0}: Error finding container 0383e2e0e6697c8e53a3580a9709fae9bc20c037491324f3abcd11f90cf19c88: Status 404 returned error can't find the container with id 0383e2e0e6697c8e53a3580a9709fae9bc20c037491324f3abcd11f90cf19c88 Sep 08 18:44:20.853727 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.853687 2553 generic.go:356] "Generic (PLEG): container finished" podID="657214e6-4d00-4d9a-9ed1-7e93621e88e5" containerID="c1afda945d0a5ba3a223e14ea67cb2ffe86d210cfa1bab5a0fb74a08a9579157" exitCode=0 Sep 08 18:44:20.854136 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.853761 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" event={"ID":"657214e6-4d00-4d9a-9ed1-7e93621e88e5","Type":"ContainerDied","Data":"c1afda945d0a5ba3a223e14ea67cb2ffe86d210cfa1bab5a0fb74a08a9579157"} Sep 08 18:44:20.854136 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.853807 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" event={"ID":"657214e6-4d00-4d9a-9ed1-7e93621e88e5","Type":"ContainerStarted","Data":"0383e2e0e6697c8e53a3580a9709fae9bc20c037491324f3abcd11f90cf19c88"} Sep 08 18:44:20.896697 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.896667 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5"] Sep 08 18:44:20.899153 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.899139 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:20.901576 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.901552 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 08 18:44:20.901707 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.901625 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-z6qp9\"" Sep 08 18:44:20.901707 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.901634 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 08 18:44:20.901707 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.901656 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 08 18:44:20.901973 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.901953 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 08 18:44:20.921265 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.921237 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5"] Sep 08 18:44:20.996860 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.996795 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21ee1433-df0e-4fb6-a807-7286c728c070-webhook-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-7tlh5\" (UID: \"21ee1433-df0e-4fb6-a807-7286c728c070\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:20.996860 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.996832 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvrv7\" (UniqueName: \"kubernetes.io/projected/21ee1433-df0e-4fb6-a807-7286c728c070-kube-api-access-nvrv7\") pod \"opendatahub-operator-controller-manager-67894858cf-7tlh5\" (UID: \"21ee1433-df0e-4fb6-a807-7286c728c070\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:20.996860 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.996868 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21ee1433-df0e-4fb6-a807-7286c728c070-tmp\") pod \"opendatahub-operator-controller-manager-67894858cf-7tlh5\" (UID: \"21ee1433-df0e-4fb6-a807-7286c728c070\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:20.997047 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.996980 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21ee1433-df0e-4fb6-a807-7286c728c070-manifests\") pod \"opendatahub-operator-controller-manager-67894858cf-7tlh5\" (UID: \"21ee1433-df0e-4fb6-a807-7286c728c070\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:20.997047 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:20.997007 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21ee1433-df0e-4fb6-a807-7286c728c070-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-7tlh5\" (UID: \"21ee1433-df0e-4fb6-a807-7286c728c070\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:21.097979 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:21.097952 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21ee1433-df0e-4fb6-a807-7286c728c070-manifests\") pod \"opendatahub-operator-controller-manager-67894858cf-7tlh5\" (UID: \"21ee1433-df0e-4fb6-a807-7286c728c070\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:21.097979 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:21.097981 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21ee1433-df0e-4fb6-a807-7286c728c070-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-7tlh5\" (UID: \"21ee1433-df0e-4fb6-a807-7286c728c070\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:21.098158 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:21.098014 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21ee1433-df0e-4fb6-a807-7286c728c070-webhook-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-7tlh5\" (UID: \"21ee1433-df0e-4fb6-a807-7286c728c070\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:21.098158 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:21.098040 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrv7\" (UniqueName: \"kubernetes.io/projected/21ee1433-df0e-4fb6-a807-7286c728c070-kube-api-access-nvrv7\") pod \"opendatahub-operator-controller-manager-67894858cf-7tlh5\" (UID: \"21ee1433-df0e-4fb6-a807-7286c728c070\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:21.098158 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:21.098070 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21ee1433-df0e-4fb6-a807-7286c728c070-tmp\") pod \"opendatahub-operator-controller-manager-67894858cf-7tlh5\" (UID: \"21ee1433-df0e-4fb6-a807-7286c728c070\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:21.098455 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:21.098433 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21ee1433-df0e-4fb6-a807-7286c728c070-manifests\") pod \"opendatahub-operator-controller-manager-67894858cf-7tlh5\" (UID: \"21ee1433-df0e-4fb6-a807-7286c728c070\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:21.100380 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:21.100343 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21ee1433-df0e-4fb6-a807-7286c728c070-tmp\") pod \"opendatahub-operator-controller-manager-67894858cf-7tlh5\" (UID: \"21ee1433-df0e-4fb6-a807-7286c728c070\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:21.100600 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:21.100582 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21ee1433-df0e-4fb6-a807-7286c728c070-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-7tlh5\" (UID: \"21ee1433-df0e-4fb6-a807-7286c728c070\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:21.100689 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:21.100625 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21ee1433-df0e-4fb6-a807-7286c728c070-webhook-cert\") pod \"opendatahub-operator-controller-manager-67894858cf-7tlh5\" (UID: \"21ee1433-df0e-4fb6-a807-7286c728c070\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:21.109037 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:21.109012 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvrv7\" (UniqueName: \"kubernetes.io/projected/21ee1433-df0e-4fb6-a807-7286c728c070-kube-api-access-nvrv7\") pod \"opendatahub-operator-controller-manager-67894858cf-7tlh5\" (UID: \"21ee1433-df0e-4fb6-a807-7286c728c070\") " pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:21.209713 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:21.209672 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:21.483653 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:21.483419 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5"] Sep 08 18:44:21.486209 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:44:21.486184 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ee1433_df0e_4fb6_a807_7286c728c070.slice/crio-da1c3a50fbaf5dc3fc6e8913ed44e836e567276ca71ffbbd10ae4e8ecca0d75b WatchSource:0}: Error finding container da1c3a50fbaf5dc3fc6e8913ed44e836e567276ca71ffbbd10ae4e8ecca0d75b: Status 404 returned error can't find the container with id da1c3a50fbaf5dc3fc6e8913ed44e836e567276ca71ffbbd10ae4e8ecca0d75b Sep 08 18:44:21.860730 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:21.860696 2553 generic.go:356] "Generic (PLEG): container finished" podID="657214e6-4d00-4d9a-9ed1-7e93621e88e5" containerID="fffe3c2c012cd19f784e32cd53847e3b56dfefe918082c6eeca4e3e9db73c87d" exitCode=0 Sep 08 18:44:21.861126 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:21.860780 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" event={"ID":"657214e6-4d00-4d9a-9ed1-7e93621e88e5","Type":"ContainerDied","Data":"fffe3c2c012cd19f784e32cd53847e3b56dfefe918082c6eeca4e3e9db73c87d"} Sep 08 18:44:21.862230 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:21.862209 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" event={"ID":"21ee1433-df0e-4fb6-a807-7286c728c070","Type":"ContainerStarted","Data":"da1c3a50fbaf5dc3fc6e8913ed44e836e567276ca71ffbbd10ae4e8ecca0d75b"} Sep 08 18:44:22.870206 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:22.870173 2553 generic.go:356] "Generic (PLEG): container finished" podID="657214e6-4d00-4d9a-9ed1-7e93621e88e5" containerID="ce83909ee991d5ee91eafa89f4a16171de3c907d2f6b5d0e6722f11c713ec873" exitCode=0 Sep 08 18:44:22.870671 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:22.870226 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" event={"ID":"657214e6-4d00-4d9a-9ed1-7e93621e88e5","Type":"ContainerDied","Data":"ce83909ee991d5ee91eafa89f4a16171de3c907d2f6b5d0e6722f11c713ec873"} Sep 08 18:44:24.481377 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.481332 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" Sep 08 18:44:24.630674 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.630649 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppn6m\" (UniqueName: \"kubernetes.io/projected/657214e6-4d00-4d9a-9ed1-7e93621e88e5-kube-api-access-ppn6m\") pod \"657214e6-4d00-4d9a-9ed1-7e93621e88e5\" (UID: \"657214e6-4d00-4d9a-9ed1-7e93621e88e5\") " Sep 08 18:44:24.631003 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.630691 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/657214e6-4d00-4d9a-9ed1-7e93621e88e5-bundle\") pod \"657214e6-4d00-4d9a-9ed1-7e93621e88e5\" (UID: \"657214e6-4d00-4d9a-9ed1-7e93621e88e5\") " Sep 08 18:44:24.631003 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.630720 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/657214e6-4d00-4d9a-9ed1-7e93621e88e5-util\") pod \"657214e6-4d00-4d9a-9ed1-7e93621e88e5\" (UID: \"657214e6-4d00-4d9a-9ed1-7e93621e88e5\") " Sep 08 18:44:24.631688 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.631658 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657214e6-4d00-4d9a-9ed1-7e93621e88e5-bundle" (OuterVolumeSpecName: "bundle") pod "657214e6-4d00-4d9a-9ed1-7e93621e88e5" (UID: "657214e6-4d00-4d9a-9ed1-7e93621e88e5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:44:24.634552 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.632844 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657214e6-4d00-4d9a-9ed1-7e93621e88e5-kube-api-access-ppn6m" (OuterVolumeSpecName: "kube-api-access-ppn6m") pod "657214e6-4d00-4d9a-9ed1-7e93621e88e5" (UID: "657214e6-4d00-4d9a-9ed1-7e93621e88e5"). InnerVolumeSpecName "kube-api-access-ppn6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:44:24.639407 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.637703 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657214e6-4d00-4d9a-9ed1-7e93621e88e5-util" (OuterVolumeSpecName: "util") pod "657214e6-4d00-4d9a-9ed1-7e93621e88e5" (UID: "657214e6-4d00-4d9a-9ed1-7e93621e88e5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:44:24.659630 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.659609 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 18:44:24.660670 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.660651 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 18:44:24.663559 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.663538 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 18:44:24.664645 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.664624 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 18:44:24.732261 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.732201 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppn6m\" (UniqueName: \"kubernetes.io/projected/657214e6-4d00-4d9a-9ed1-7e93621e88e5-kube-api-access-ppn6m\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:44:24.732261 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.732233 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/657214e6-4d00-4d9a-9ed1-7e93621e88e5-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:44:24.732261 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.732249 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/657214e6-4d00-4d9a-9ed1-7e93621e88e5-util\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:44:24.882135 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.882109 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" Sep 08 18:44:24.882333 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.882110 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0pkrh6" event={"ID":"657214e6-4d00-4d9a-9ed1-7e93621e88e5","Type":"ContainerDied","Data":"0383e2e0e6697c8e53a3580a9709fae9bc20c037491324f3abcd11f90cf19c88"} Sep 08 18:44:24.882333 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.882236 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0383e2e0e6697c8e53a3580a9709fae9bc20c037491324f3abcd11f90cf19c88" Sep 08 18:44:24.883705 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:24.883678 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" event={"ID":"21ee1433-df0e-4fb6-a807-7286c728c070","Type":"ContainerStarted","Data":"7416f851bb5be163e95611b0a7caab7751ec5e7a25762b91538dab4108274af2"} Sep 08 18:44:25.163537 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:25.163506 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88f8c8bfb-q9rdw"] Sep 08 18:44:25.833819 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:25.833790 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68bb56fbd7-kw7fb" Sep 08 18:44:25.890200 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:25.890169 2553 generic.go:356] "Generic (PLEG): container finished" podID="21ee1433-df0e-4fb6-a807-7286c728c070" containerID="7416f851bb5be163e95611b0a7caab7751ec5e7a25762b91538dab4108274af2" exitCode=0 Sep 08 18:44:25.890358 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:25.890251 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" event={"ID":"21ee1433-df0e-4fb6-a807-7286c728c070","Type":"ContainerDied","Data":"7416f851bb5be163e95611b0a7caab7751ec5e7a25762b91538dab4108274af2"} Sep 08 18:44:26.897207 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:26.897170 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" event={"ID":"21ee1433-df0e-4fb6-a807-7286c728c070","Type":"ContainerStarted","Data":"1c0276e962bcf1294c0f31f17c3de9ce7473cb97dcfe2c32f1709b726fbc06c3"} Sep 08 18:44:26.897629 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:26.897310 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:26.920602 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:26.920555 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" podStartSLOduration=2.205694638 podStartE2EDuration="6.920540322s" totalImagesPullingTime="3.570555949s" totalInitContainerRuntime="1.144289735s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:44:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:44:26.918452685 +0000 UTC m=+602.624340917" watchObservedRunningTime="2026-09-08 18:44:26.920540322 +0000 UTC m=+602.626428554" Sep 08 18:44:37.904962 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:37.904933 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67894858cf-7tlh5" Sep 08 18:44:50.183684 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.183599 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-88f8c8bfb-q9rdw" podUID="024643c8-804d-4e3e-9141-1e545cb482dc" containerName="console" containerID="cri-o://88a8573473936aa96987a9e46a6208735c5fb19b126b5ff372061bacbb026702" gracePeriod=15 Sep 08 18:44:50.289679 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.289642 2553 patch_prober.go:28] interesting pod/console-88f8c8bfb-q9rdw container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.22:8443/health\": dial tcp 10.134.0.22:8443: connect: connection refused" start-of-body= Sep 08 18:44:50.289836 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.289697 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-88f8c8bfb-q9rdw" podUID="024643c8-804d-4e3e-9141-1e545cb482dc" containerName="console" probeResult="failure" output="Get \"https://10.134.0.22:8443/health\": dial tcp 10.134.0.22:8443: connect: connection refused" Sep 08 18:44:50.419027 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.419002 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:44:50.431938 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.431917 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-console-config\") pod \"024643c8-804d-4e3e-9141-1e545cb482dc\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " Sep 08 18:44:50.432052 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.431946 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-oauth-serving-cert\") pod \"024643c8-804d-4e3e-9141-1e545cb482dc\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " Sep 08 18:44:50.432052 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.431976 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-service-ca\") pod \"024643c8-804d-4e3e-9141-1e545cb482dc\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " Sep 08 18:44:50.432052 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.432000 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh8pk\" (UniqueName: \"kubernetes.io/projected/024643c8-804d-4e3e-9141-1e545cb482dc-kube-api-access-zh8pk\") pod \"024643c8-804d-4e3e-9141-1e545cb482dc\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " Sep 08 18:44:50.432052 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.432019 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/024643c8-804d-4e3e-9141-1e545cb482dc-console-oauth-config\") pod \"024643c8-804d-4e3e-9141-1e545cb482dc\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " Sep 08 18:44:50.432268 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.432074 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-trusted-ca-bundle\") pod \"024643c8-804d-4e3e-9141-1e545cb482dc\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " Sep 08 18:44:50.432268 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.432122 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/024643c8-804d-4e3e-9141-1e545cb482dc-console-serving-cert\") pod \"024643c8-804d-4e3e-9141-1e545cb482dc\" (UID: \"024643c8-804d-4e3e-9141-1e545cb482dc\") " Sep 08 18:44:50.432396 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.432261 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-console-config" (OuterVolumeSpecName: "console-config") pod "024643c8-804d-4e3e-9141-1e545cb482dc" (UID: "024643c8-804d-4e3e-9141-1e545cb482dc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:44:50.432396 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.432347 2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-console-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:44:50.432522 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.432411 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "024643c8-804d-4e3e-9141-1e545cb482dc" (UID: "024643c8-804d-4e3e-9141-1e545cb482dc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:44:50.432574 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.432530 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-service-ca" (OuterVolumeSpecName: "service-ca") pod "024643c8-804d-4e3e-9141-1e545cb482dc" (UID: "024643c8-804d-4e3e-9141-1e545cb482dc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:44:50.432614 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.432601 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "024643c8-804d-4e3e-9141-1e545cb482dc" (UID: "024643c8-804d-4e3e-9141-1e545cb482dc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:44:50.434200 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.434154 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024643c8-804d-4e3e-9141-1e545cb482dc-kube-api-access-zh8pk" (OuterVolumeSpecName: "kube-api-access-zh8pk") pod "024643c8-804d-4e3e-9141-1e545cb482dc" (UID: "024643c8-804d-4e3e-9141-1e545cb482dc"). InnerVolumeSpecName "kube-api-access-zh8pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:44:50.434293 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.434200 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024643c8-804d-4e3e-9141-1e545cb482dc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "024643c8-804d-4e3e-9141-1e545cb482dc" (UID: "024643c8-804d-4e3e-9141-1e545cb482dc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:44:50.434293 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.434217 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024643c8-804d-4e3e-9141-1e545cb482dc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "024643c8-804d-4e3e-9141-1e545cb482dc" (UID: "024643c8-804d-4e3e-9141-1e545cb482dc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:44:50.533551 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.533518 2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-oauth-serving-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:44:50.533551 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.533545 2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:44:50.533551 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.533555 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh8pk\" (UniqueName: \"kubernetes.io/projected/024643c8-804d-4e3e-9141-1e545cb482dc-kube-api-access-zh8pk\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:44:50.533758 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.533564 2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/024643c8-804d-4e3e-9141-1e545cb482dc-console-oauth-config\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:44:50.533758 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.533573 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/024643c8-804d-4e3e-9141-1e545cb482dc-trusted-ca-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:44:50.533758 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:50.533582 2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/024643c8-804d-4e3e-9141-1e545cb482dc-console-serving-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:44:51.004121 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.004029 2553 generic.go:356] "Generic (PLEG): container finished" podID="024643c8-804d-4e3e-9141-1e545cb482dc" containerID="88a8573473936aa96987a9e46a6208735c5fb19b126b5ff372061bacbb026702" exitCode=0 Sep 08 18:44:51.004121 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.004072 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88f8c8bfb-q9rdw" Sep 08 18:44:51.004349 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.004111 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88f8c8bfb-q9rdw" event={"ID":"024643c8-804d-4e3e-9141-1e545cb482dc","Type":"ContainerDied","Data":"88a8573473936aa96987a9e46a6208735c5fb19b126b5ff372061bacbb026702"} Sep 08 18:44:51.004349 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.004170 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88f8c8bfb-q9rdw" event={"ID":"024643c8-804d-4e3e-9141-1e545cb482dc","Type":"ContainerDied","Data":"286b40d492e05ad9e1c27250f11b538144c64c8a7c372be70bf06be1a78ca384"} Sep 08 18:44:51.004349 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.004198 2553 scope.go:117] "RemoveContainer" containerID="88a8573473936aa96987a9e46a6208735c5fb19b126b5ff372061bacbb026702" Sep 08 18:44:51.012631 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.012583 2553 scope.go:117] "RemoveContainer" containerID="88a8573473936aa96987a9e46a6208735c5fb19b126b5ff372061bacbb026702" Sep 08 18:44:51.012885 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:44:51.012864 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a8573473936aa96987a9e46a6208735c5fb19b126b5ff372061bacbb026702\": container with ID starting with 88a8573473936aa96987a9e46a6208735c5fb19b126b5ff372061bacbb026702 not found: ID does not exist" containerID="88a8573473936aa96987a9e46a6208735c5fb19b126b5ff372061bacbb026702" Sep 08 18:44:51.012938 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.012893 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a8573473936aa96987a9e46a6208735c5fb19b126b5ff372061bacbb026702"} err="failed to get container status \"88a8573473936aa96987a9e46a6208735c5fb19b126b5ff372061bacbb026702\": rpc error: code = NotFound desc = could not find container \"88a8573473936aa96987a9e46a6208735c5fb19b126b5ff372061bacbb026702\": container with ID starting with 88a8573473936aa96987a9e46a6208735c5fb19b126b5ff372061bacbb026702 not found: ID does not exist" Sep 08 18:44:51.023133 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.023109 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88f8c8bfb-q9rdw"] Sep 08 18:44:51.026813 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.026791 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-88f8c8bfb-q9rdw"] Sep 08 18:44:51.791014 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.790981 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wtnf6"] Sep 08 18:44:51.791355 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.791288 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="024643c8-804d-4e3e-9141-1e545cb482dc" containerName="console" Sep 08 18:44:51.791355 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.791299 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="024643c8-804d-4e3e-9141-1e545cb482dc" containerName="console" Sep 08 18:44:51.791355 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.791307 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657214e6-4d00-4d9a-9ed1-7e93621e88e5" containerName="util" Sep 08 18:44:51.791355 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.791313 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="657214e6-4d00-4d9a-9ed1-7e93621e88e5" containerName="util" Sep 08 18:44:51.791355 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.791320 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657214e6-4d00-4d9a-9ed1-7e93621e88e5" containerName="pull" Sep 08 18:44:51.791355 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.791325 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="657214e6-4d00-4d9a-9ed1-7e93621e88e5" containerName="pull" Sep 08 18:44:51.791355 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.791334 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="657214e6-4d00-4d9a-9ed1-7e93621e88e5" containerName="extract" Sep 08 18:44:51.791355 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.791339 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="657214e6-4d00-4d9a-9ed1-7e93621e88e5" containerName="extract" Sep 08 18:44:51.791638 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.791402 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="657214e6-4d00-4d9a-9ed1-7e93621e88e5" containerName="extract" Sep 08 18:44:51.791638 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.791410 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="024643c8-804d-4e3e-9141-1e545cb482dc" containerName="console" Sep 08 18:44:51.795692 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.795672 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wtnf6" Sep 08 18:44:51.797985 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.797961 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 08 18:44:51.798096 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.798076 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2ncgk\"" Sep 08 18:44:51.801651 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.801627 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wtnf6"] Sep 08 18:44:51.846495 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.846450 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x257\" (UniqueName: \"kubernetes.io/projected/603bb783-22ba-4bfb-a0c5-82adf214f386-kube-api-access-7x257\") pod \"odh-model-controller-7998c67dc4-wtnf6\" (UID: \"603bb783-22ba-4bfb-a0c5-82adf214f386\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtnf6" Sep 08 18:44:51.846683 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.846504 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/603bb783-22ba-4bfb-a0c5-82adf214f386-cert\") pod \"odh-model-controller-7998c67dc4-wtnf6\" (UID: \"603bb783-22ba-4bfb-a0c5-82adf214f386\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtnf6" Sep 08 18:44:51.947327 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.947287 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x257\" (UniqueName: \"kubernetes.io/projected/603bb783-22ba-4bfb-a0c5-82adf214f386-kube-api-access-7x257\") pod \"odh-model-controller-7998c67dc4-wtnf6\" (UID: \"603bb783-22ba-4bfb-a0c5-82adf214f386\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtnf6" Sep 08 18:44:51.947513 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.947339 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/603bb783-22ba-4bfb-a0c5-82adf214f386-cert\") pod \"odh-model-controller-7998c67dc4-wtnf6\" (UID: \"603bb783-22ba-4bfb-a0c5-82adf214f386\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtnf6" Sep 08 18:44:51.947588 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:44:51.947511 2553 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 08 18:44:51.947640 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:44:51.947590 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/603bb783-22ba-4bfb-a0c5-82adf214f386-cert podName:603bb783-22ba-4bfb-a0c5-82adf214f386 nodeName:}" failed. No retries permitted until 2026-09-08 18:44:52.447571068 +0000 UTC m=+628.153459298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/603bb783-22ba-4bfb-a0c5-82adf214f386-cert") pod "odh-model-controller-7998c67dc4-wtnf6" (UID: "603bb783-22ba-4bfb-a0c5-82adf214f386") : secret "odh-model-controller-webhook-cert" not found Sep 08 18:44:51.958043 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:51.958001 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x257\" (UniqueName: \"kubernetes.io/projected/603bb783-22ba-4bfb-a0c5-82adf214f386-kube-api-access-7x257\") pod \"odh-model-controller-7998c67dc4-wtnf6\" (UID: \"603bb783-22ba-4bfb-a0c5-82adf214f386\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtnf6" Sep 08 18:44:52.334981 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.334945 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-m7qsp"] Sep 08 18:44:52.338359 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.338335 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-m7qsp" Sep 08 18:44:52.340946 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.340925 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 08 18:44:52.341052 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.340975 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-b2sbq\"" Sep 08 18:44:52.349636 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.349604 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-m7qsp"] Sep 08 18:44:52.351337 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.351313 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnnr4\" (UniqueName: \"kubernetes.io/projected/c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef-kube-api-access-wnnr4\") pod \"model-serving-api-54d89bd5fc-m7qsp\" (UID: \"c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef\") " pod="opendatahub/model-serving-api-54d89bd5fc-m7qsp" Sep 08 18:44:52.351464 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.351405 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef-tls-certs\") pod \"model-serving-api-54d89bd5fc-m7qsp\" (UID: \"c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef\") " pod="opendatahub/model-serving-api-54d89bd5fc-m7qsp" Sep 08 18:44:52.452194 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.452162 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/603bb783-22ba-4bfb-a0c5-82adf214f386-cert\") pod \"odh-model-controller-7998c67dc4-wtnf6\" (UID: \"603bb783-22ba-4bfb-a0c5-82adf214f386\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtnf6" Sep 08 18:44:52.452388 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.452198 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnnr4\" (UniqueName: \"kubernetes.io/projected/c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef-kube-api-access-wnnr4\") pod \"model-serving-api-54d89bd5fc-m7qsp\" (UID: \"c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef\") " pod="opendatahub/model-serving-api-54d89bd5fc-m7qsp" Sep 08 18:44:52.452388 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.452238 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef-tls-certs\") pod \"model-serving-api-54d89bd5fc-m7qsp\" (UID: \"c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef\") " pod="opendatahub/model-serving-api-54d89bd5fc-m7qsp" Sep 08 18:44:52.454845 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.454816 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef-tls-certs\") pod \"model-serving-api-54d89bd5fc-m7qsp\" (UID: \"c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef\") " pod="opendatahub/model-serving-api-54d89bd5fc-m7qsp" Sep 08 18:44:52.454845 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.454837 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/603bb783-22ba-4bfb-a0c5-82adf214f386-cert\") pod \"odh-model-controller-7998c67dc4-wtnf6\" (UID: \"603bb783-22ba-4bfb-a0c5-82adf214f386\") " pod="opendatahub/odh-model-controller-7998c67dc4-wtnf6" Sep 08 18:44:52.459980 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.459955 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnnr4\" (UniqueName: \"kubernetes.io/projected/c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef-kube-api-access-wnnr4\") pod \"model-serving-api-54d89bd5fc-m7qsp\" (UID: \"c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef\") " pod="opendatahub/model-serving-api-54d89bd5fc-m7qsp" Sep 08 18:44:52.650170 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.650130 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-m7qsp" Sep 08 18:44:52.707673 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.707634 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wtnf6" Sep 08 18:44:52.747502 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.747454 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="024643c8-804d-4e3e-9141-1e545cb482dc" path="/var/lib/kubelet/pods/024643c8-804d-4e3e-9141-1e545cb482dc/volumes" Sep 08 18:44:52.925990 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.925921 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-m7qsp"] Sep 08 18:44:52.928454 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:44:52.928412 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d68d3a_cb6a_4d3a_96b1_3b7a9d162aef.slice/crio-fbc00b67ac6ed7366d68647ae15037169e93e2d474a01d8408f1db3015999304 WatchSource:0}: Error finding container fbc00b67ac6ed7366d68647ae15037169e93e2d474a01d8408f1db3015999304: Status 404 returned error can't find the container with id fbc00b67ac6ed7366d68647ae15037169e93e2d474a01d8408f1db3015999304 Sep 08 18:44:52.951557 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.951517 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z"] Sep 08 18:44:52.956801 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.956773 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:52.959446 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.959337 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 18:44:52.959876 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.959589 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 08 18:44:52.959876 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.959702 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 08 18:44:52.959876 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.959589 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-g8cwg\"" Sep 08 18:44:52.971519 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.971488 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z"] Sep 08 18:44:52.991100 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:52.991075 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wtnf6"] Sep 08 18:44:52.994066 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:44:52.994036 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603bb783_22ba_4bfb_a0c5_82adf214f386.slice/crio-9909b2024ebcf2827fbf9bae1b38df3e0cf0c626e51ed96351cf418e2842993c WatchSource:0}: Error finding container 9909b2024ebcf2827fbf9bae1b38df3e0cf0c626e51ed96351cf418e2842993c: Status 404 returned error can't find the container with id 9909b2024ebcf2827fbf9bae1b38df3e0cf0c626e51ed96351cf418e2842993c Sep 08 18:44:53.025121 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.025082 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wtnf6" event={"ID":"603bb783-22ba-4bfb-a0c5-82adf214f386","Type":"ContainerStarted","Data":"9909b2024ebcf2827fbf9bae1b38df3e0cf0c626e51ed96351cf418e2842993c"} Sep 08 18:44:53.026288 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.026265 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-m7qsp" event={"ID":"c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef","Type":"ContainerStarted","Data":"fbc00b67ac6ed7366d68647ae15037169e93e2d474a01d8408f1db3015999304"} Sep 08 18:44:53.056350 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.056314 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pz8c\" (UniqueName: \"kubernetes.io/projected/c29c6702-c2f2-4226-9f37-92680e2bdd62-kube-api-access-8pz8c\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.056511 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.056390 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29c6702-c2f2-4226-9f37-92680e2bdd62-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.056511 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.056419 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29c6702-c2f2-4226-9f37-92680e2bdd62-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.056511 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.056442 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c6702-c2f2-4226-9f37-92680e2bdd62-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.056511 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.056461 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c6702-c2f2-4226-9f37-92680e2bdd62-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.056511 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.056496 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29c6702-c2f2-4226-9f37-92680e2bdd62-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.056746 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.056548 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29c6702-c2f2-4226-9f37-92680e2bdd62-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.056746 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.056628 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29c6702-c2f2-4226-9f37-92680e2bdd62-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.056746 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.056670 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29c6702-c2f2-4226-9f37-92680e2bdd62-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.157703 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.157666 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29c6702-c2f2-4226-9f37-92680e2bdd62-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.157703 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.157706 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29c6702-c2f2-4226-9f37-92680e2bdd62-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.157949 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.157730 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c6702-c2f2-4226-9f37-92680e2bdd62-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.157949 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.157767 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c6702-c2f2-4226-9f37-92680e2bdd62-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.157949 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.157790 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29c6702-c2f2-4226-9f37-92680e2bdd62-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.157949 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.157808 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29c6702-c2f2-4226-9f37-92680e2bdd62-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.157949 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.157831 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29c6702-c2f2-4226-9f37-92680e2bdd62-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.157949 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.157861 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29c6702-c2f2-4226-9f37-92680e2bdd62-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.157949 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.157917 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pz8c\" (UniqueName: \"kubernetes.io/projected/c29c6702-c2f2-4226-9f37-92680e2bdd62-kube-api-access-8pz8c\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.158308 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.158197 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c29c6702-c2f2-4226-9f37-92680e2bdd62-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.158308 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.158246 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c6702-c2f2-4226-9f37-92680e2bdd62-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.158579 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.158553 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c29c6702-c2f2-4226-9f37-92680e2bdd62-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.158701 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.158612 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c29c6702-c2f2-4226-9f37-92680e2bdd62-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.158701 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.158672 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c29c6702-c2f2-4226-9f37-92680e2bdd62-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.160211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.160188 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c29c6702-c2f2-4226-9f37-92680e2bdd62-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.161021 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.160996 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c29c6702-c2f2-4226-9f37-92680e2bdd62-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.165290 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.165267 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c29c6702-c2f2-4226-9f37-92680e2bdd62-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.166036 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.166008 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pz8c\" (UniqueName: \"kubernetes.io/projected/c29c6702-c2f2-4226-9f37-92680e2bdd62-kube-api-access-8pz8c\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z\" (UID: \"c29c6702-c2f2-4226-9f37-92680e2bdd62\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.284317 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.284240 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:53.561037 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:53.560992 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z"] Sep 08 18:44:53.563774 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:44:53.563742 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29c6702_c2f2_4226_9f37_92680e2bdd62.slice/crio-adb68504361d216af54b30fbfe9893f44bcb9cd131a59024afd3313054678f2c WatchSource:0}: Error finding container adb68504361d216af54b30fbfe9893f44bcb9cd131a59024afd3313054678f2c: Status 404 returned error can't find the container with id adb68504361d216af54b30fbfe9893f44bcb9cd131a59024afd3313054678f2c Sep 08 18:44:54.041639 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:54.039335 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" event={"ID":"c29c6702-c2f2-4226-9f37-92680e2bdd62","Type":"ContainerStarted","Data":"adb68504361d216af54b30fbfe9893f44bcb9cd131a59024afd3313054678f2c"} Sep 08 18:44:58.606896 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:58.606854 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:44:58.607156 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:58.606954 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:44:58.607156 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:58.606996 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:44:59.073856 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.073814 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-m7qsp" event={"ID":"c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef","Type":"ContainerStarted","Data":"88483c4f9c6aa45e5e5e0724d52427a15ecc4b051cc6d4e49c54fd6753f60245"} Sep 08 18:44:59.074069 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.073951 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-m7qsp" Sep 08 18:44:59.075648 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.075616 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" event={"ID":"c29c6702-c2f2-4226-9f37-92680e2bdd62","Type":"ContainerStarted","Data":"e7abd126d5c61b5ea21c4884f1b0ca4557cbb0c687657a763f7e35d685fca50c"} Sep 08 18:44:59.077184 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.077161 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wtnf6" event={"ID":"603bb783-22ba-4bfb-a0c5-82adf214f386","Type":"ContainerStarted","Data":"ea8acb5360893fe68b5091ad8d202767c3837b68e8fb276b37dd192879b320c3"} Sep 08 18:44:59.077320 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.077302 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-wtnf6" Sep 08 18:44:59.095908 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.095868 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-m7qsp" podStartSLOduration=1.483547926 podStartE2EDuration="7.095856145s" totalImagesPullingTime="5.612308219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:44:59.094330953 +0000 UTC m=+634.800219186" watchObservedRunningTime="2026-09-08 18:44:59.095856145 +0000 UTC m=+634.801744378" Sep 08 18:44:59.110864 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.110825 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-wtnf6" podStartSLOduration=2.552480552 podStartE2EDuration="8.110814426s" totalImagesPullingTime="5.558333874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:44:59.108814826 +0000 UTC m=+634.814703061" watchObservedRunningTime="2026-09-08 18:44:59.110814426 +0000 UTC m=+634.816702701" Sep 08 18:44:59.128670 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.128622 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" podStartSLOduration=2.088370872 podStartE2EDuration="7.128606216s" totalImagesPullingTime="5.040235344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:44:59.127692505 +0000 UTC m=+634.833580741" watchObservedRunningTime="2026-09-08 18:44:59.128606216 +0000 UTC m=+634.834494449" Sep 08 18:44:59.285075 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.285037 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:59.297344 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.290728 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:44:59.734397 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.734349 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm"] Sep 08 18:44:59.737652 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.737633 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.740269 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.740248 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 08 18:44:59.740412 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.740277 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 08 18:44:59.740412 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.740282 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tpvnd\"" Sep 08 18:44:59.740583 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.740569 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 08 18:44:59.748118 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.748096 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm"] Sep 08 18:44:59.820454 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.820421 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4xqb\" (UniqueName: \"kubernetes.io/projected/bc171b62-76f6-462a-8281-b3aa288c422c-kube-api-access-h4xqb\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.820586 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.820477 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc171b62-76f6-462a-8281-b3aa288c422c-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.820586 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.820496 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc171b62-76f6-462a-8281-b3aa288c422c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.820679 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.820588 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc171b62-76f6-462a-8281-b3aa288c422c-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.820679 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.820626 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bc171b62-76f6-462a-8281-b3aa288c422c-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.820823 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.820808 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc171b62-76f6-462a-8281-b3aa288c422c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.820874 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.820861 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc171b62-76f6-462a-8281-b3aa288c422c-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.921924 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.921895 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc171b62-76f6-462a-8281-b3aa288c422c-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.921924 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.921926 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc171b62-76f6-462a-8281-b3aa288c422c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.922156 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.921950 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc171b62-76f6-462a-8281-b3aa288c422c-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.922156 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.921968 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bc171b62-76f6-462a-8281-b3aa288c422c-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.922156 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.922020 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc171b62-76f6-462a-8281-b3aa288c422c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.922156 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.922055 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc171b62-76f6-462a-8281-b3aa288c422c-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.922350 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.922228 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4xqb\" (UniqueName: \"kubernetes.io/projected/bc171b62-76f6-462a-8281-b3aa288c422c-kube-api-access-h4xqb\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.922849 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.922824 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bc171b62-76f6-462a-8281-b3aa288c422c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.924419 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.924398 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bc171b62-76f6-462a-8281-b3aa288c422c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.924530 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.924447 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bc171b62-76f6-462a-8281-b3aa288c422c-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.925229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.925199 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc171b62-76f6-462a-8281-b3aa288c422c-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.925321 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.925212 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bc171b62-76f6-462a-8281-b3aa288c422c-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.929651 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.929631 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc171b62-76f6-462a-8281-b3aa288c422c-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:44:59.930030 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:44:59.930004 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4xqb\" (UniqueName: \"kubernetes.io/projected/bc171b62-76f6-462a-8281-b3aa288c422c-kube-api-access-h4xqb\") pod \"istiod-openshift-gateway-5677ccb49b-czckm\" (UID: \"bc171b62-76f6-462a-8281-b3aa288c422c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:45:00.047465 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:00.047349 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:45:00.084822 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:00.084763 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:45:00.086129 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:00.086099 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z" Sep 08 18:45:00.332775 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:00.332323 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm"] Sep 08 18:45:00.337794 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:45:00.337765 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc171b62_76f6_462a_8281_b3aa288c422c.slice/crio-7dc1e8cc5b55d4d000fbb09bddf3b8b10678d43b79092c4be2df529226454351 WatchSource:0}: Error finding container 7dc1e8cc5b55d4d000fbb09bddf3b8b10678d43b79092c4be2df529226454351: Status 404 returned error can't find the container with id 7dc1e8cc5b55d4d000fbb09bddf3b8b10678d43b79092c4be2df529226454351 Sep 08 18:45:01.089401 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:01.089343 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" event={"ID":"bc171b62-76f6-462a-8281-b3aa288c422c","Type":"ContainerStarted","Data":"7dc1e8cc5b55d4d000fbb09bddf3b8b10678d43b79092c4be2df529226454351"} Sep 08 18:45:02.940806 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:02.940768 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:45:02.941077 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:02.940839 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:45:03.101421 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:03.101389 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" event={"ID":"bc171b62-76f6-462a-8281-b3aa288c422c","Type":"ContainerStarted","Data":"803e342116e8c39eaac41fd09f289884d3026de52fbc38ccef6edefcd2142e22"} Sep 08 18:45:03.101563 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:03.101543 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:45:03.103039 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:03.103014 2553 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-czckm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 18:45:03.103158 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:03.103060 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" podUID="bc171b62-76f6-462a-8281-b3aa288c422c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 18:45:03.137710 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:03.137658 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" podStartSLOduration=1.537685748 podStartE2EDuration="4.137643823s" totalImagesPullingTime="2.599958075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:45:03.13488553 +0000 UTC m=+638.840773763" watchObservedRunningTime="2026-09-08 18:45:03.137643823 +0000 UTC m=+638.843532054" Sep 08 18:45:04.106834 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:04.106798 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-czckm" Sep 08 18:45:10.086072 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:10.086030 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-wtnf6" Sep 08 18:45:10.089310 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:10.089287 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-m7qsp" Sep 08 18:45:56.408062 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.408027 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c"] Sep 08 18:45:56.410533 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.410513 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" Sep 08 18:45:56.413197 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.413176 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f66nj\"" Sep 08 18:45:56.413197 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.413184 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 18:45:56.414116 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.414097 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 18:45:56.419538 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.419514 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c"] Sep 08 18:45:56.494889 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.494856 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb014585-9880-469b-9ed8-a22971997cca-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c\" (UID: \"cb014585-9880-469b-9ed8-a22971997cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" Sep 08 18:45:56.495063 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.494932 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f69w4\" (UniqueName: \"kubernetes.io/projected/cb014585-9880-469b-9ed8-a22971997cca-kube-api-access-f69w4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c\" (UID: \"cb014585-9880-469b-9ed8-a22971997cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" Sep 08 18:45:56.495063 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.494961 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb014585-9880-469b-9ed8-a22971997cca-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c\" (UID: \"cb014585-9880-469b-9ed8-a22971997cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" Sep 08 18:45:56.507487 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.507461 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr"] Sep 08 18:45:56.509741 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.509727 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" Sep 08 18:45:56.518644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.518622 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr"] Sep 08 18:45:56.596073 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.596045 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmpdv\" (UniqueName: \"kubernetes.io/projected/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-kube-api-access-gmpdv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr\" (UID: \"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" Sep 08 18:45:56.596213 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.596087 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb014585-9880-469b-9ed8-a22971997cca-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c\" (UID: \"cb014585-9880-469b-9ed8-a22971997cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" Sep 08 18:45:56.596213 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.596170 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr\" (UID: \"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" Sep 08 18:45:56.596299 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.596213 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f69w4\" (UniqueName: \"kubernetes.io/projected/cb014585-9880-469b-9ed8-a22971997cca-kube-api-access-f69w4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c\" (UID: \"cb014585-9880-469b-9ed8-a22971997cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" Sep 08 18:45:56.596299 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.596238 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb014585-9880-469b-9ed8-a22971997cca-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c\" (UID: \"cb014585-9880-469b-9ed8-a22971997cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" Sep 08 18:45:56.596299 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.596274 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr\" (UID: \"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" Sep 08 18:45:56.596524 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.596505 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb014585-9880-469b-9ed8-a22971997cca-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c\" (UID: \"cb014585-9880-469b-9ed8-a22971997cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" Sep 08 18:45:56.596587 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.596571 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb014585-9880-469b-9ed8-a22971997cca-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c\" (UID: \"cb014585-9880-469b-9ed8-a22971997cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" Sep 08 18:45:56.604121 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.604094 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257"] Sep 08 18:45:56.606525 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.606502 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f69w4\" (UniqueName: \"kubernetes.io/projected/cb014585-9880-469b-9ed8-a22971997cca-kube-api-access-f69w4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c\" (UID: \"cb014585-9880-469b-9ed8-a22971997cca\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" Sep 08 18:45:56.606642 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.606628 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" Sep 08 18:45:56.615021 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.614997 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257"] Sep 08 18:45:56.697329 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.697244 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rt47\" (UniqueName: \"kubernetes.io/projected/035d85c9-1c2d-49f7-89b1-7224285292a3-kube-api-access-4rt47\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257\" (UID: \"035d85c9-1c2d-49f7-89b1-7224285292a3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" Sep 08 18:45:56.697329 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.697289 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr\" (UID: \"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" Sep 08 18:45:56.697329 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.697306 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/035d85c9-1c2d-49f7-89b1-7224285292a3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257\" (UID: \"035d85c9-1c2d-49f7-89b1-7224285292a3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" Sep 08 18:45:56.697626 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.697467 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmpdv\" (UniqueName: \"kubernetes.io/projected/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-kube-api-access-gmpdv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr\" (UID: \"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" Sep 08 18:45:56.697626 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.697520 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/035d85c9-1c2d-49f7-89b1-7224285292a3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257\" (UID: \"035d85c9-1c2d-49f7-89b1-7224285292a3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" Sep 08 18:45:56.697626 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.697606 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr\" (UID: \"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" Sep 08 18:45:56.697809 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.697693 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr\" (UID: \"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" Sep 08 18:45:56.697979 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.697951 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr\" (UID: \"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" Sep 08 18:45:56.708958 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.708937 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd"] Sep 08 18:45:56.710317 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.710293 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmpdv\" (UniqueName: \"kubernetes.io/projected/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-kube-api-access-gmpdv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr\" (UID: \"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" Sep 08 18:45:56.711754 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.711740 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" Sep 08 18:45:56.720291 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.720269 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" Sep 08 18:45:56.720411 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.720395 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd"] Sep 08 18:45:56.798512 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.798480 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5fvj\" (UniqueName: \"kubernetes.io/projected/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-kube-api-access-h5fvj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd\" (UID: \"3fcd8c72-a05b-4b74-a88a-ff874fa717b6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" Sep 08 18:45:56.798634 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.798544 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rt47\" (UniqueName: \"kubernetes.io/projected/035d85c9-1c2d-49f7-89b1-7224285292a3-kube-api-access-4rt47\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257\" (UID: \"035d85c9-1c2d-49f7-89b1-7224285292a3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" Sep 08 18:45:56.798674 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.798652 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/035d85c9-1c2d-49f7-89b1-7224285292a3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257\" (UID: \"035d85c9-1c2d-49f7-89b1-7224285292a3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" Sep 08 18:45:56.798719 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.798681 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd\" (UID: \"3fcd8c72-a05b-4b74-a88a-ff874fa717b6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" Sep 08 18:45:56.798719 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.798712 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd\" (UID: \"3fcd8c72-a05b-4b74-a88a-ff874fa717b6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" Sep 08 18:45:56.798926 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.798904 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/035d85c9-1c2d-49f7-89b1-7224285292a3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257\" (UID: \"035d85c9-1c2d-49f7-89b1-7224285292a3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" Sep 08 18:45:56.799071 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.799053 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/035d85c9-1c2d-49f7-89b1-7224285292a3-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257\" (UID: \"035d85c9-1c2d-49f7-89b1-7224285292a3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" Sep 08 18:45:56.799174 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.799158 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/035d85c9-1c2d-49f7-89b1-7224285292a3-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257\" (UID: \"035d85c9-1c2d-49f7-89b1-7224285292a3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" Sep 08 18:45:56.809490 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.809469 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rt47\" (UniqueName: \"kubernetes.io/projected/035d85c9-1c2d-49f7-89b1-7224285292a3-kube-api-access-4rt47\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257\" (UID: \"035d85c9-1c2d-49f7-89b1-7224285292a3\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" Sep 08 18:45:56.819269 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.819250 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" Sep 08 18:45:56.900172 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.900137 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5fvj\" (UniqueName: \"kubernetes.io/projected/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-kube-api-access-h5fvj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd\" (UID: \"3fcd8c72-a05b-4b74-a88a-ff874fa717b6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" Sep 08 18:45:56.900317 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.900262 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd\" (UID: \"3fcd8c72-a05b-4b74-a88a-ff874fa717b6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" Sep 08 18:45:56.900317 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.900309 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd\" (UID: \"3fcd8c72-a05b-4b74-a88a-ff874fa717b6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" Sep 08 18:45:56.900687 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.900657 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd\" (UID: \"3fcd8c72-a05b-4b74-a88a-ff874fa717b6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" Sep 08 18:45:56.900765 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.900674 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd\" (UID: \"3fcd8c72-a05b-4b74-a88a-ff874fa717b6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" Sep 08 18:45:56.908996 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.908974 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5fvj\" (UniqueName: \"kubernetes.io/projected/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-kube-api-access-h5fvj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd\" (UID: \"3fcd8c72-a05b-4b74-a88a-ff874fa717b6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" Sep 08 18:45:56.922521 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.922490 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" Sep 08 18:45:56.994884 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:56.994853 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c"] Sep 08 18:45:56.995571 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:45:56.995541 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb014585_9880_469b_9ed8_a22971997cca.slice/crio-ffb55e1b093d1bc6bea5dc8d5501dc39861cc6d92a00f7f0189f1ea8adde4d88 WatchSource:0}: Error finding container ffb55e1b093d1bc6bea5dc8d5501dc39861cc6d92a00f7f0189f1ea8adde4d88: Status 404 returned error can't find the container with id ffb55e1b093d1bc6bea5dc8d5501dc39861cc6d92a00f7f0189f1ea8adde4d88 Sep 08 18:45:57.027037 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:57.027012 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" Sep 08 18:45:57.093721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:57.093684 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr"] Sep 08 18:45:57.102954 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:45:57.102910 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b999ed_8ab7_46e0_8c1d_1f465e5666ee.slice/crio-671e7a0598f7b750c0bc6e90e2049ef45458d0c5c73e6c91da681619e801b699 WatchSource:0}: Error finding container 671e7a0598f7b750c0bc6e90e2049ef45458d0c5c73e6c91da681619e801b699: Status 404 returned error can't find the container with id 671e7a0598f7b750c0bc6e90e2049ef45458d0c5c73e6c91da681619e801b699 Sep 08 18:45:57.194580 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:57.194554 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257"] Sep 08 18:45:57.195994 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:45:57.195965 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod035d85c9_1c2d_49f7_89b1_7224285292a3.slice/crio-fbfbad33f192567fb13c3ffca18e0ea060be189375e09c8bdef753ded3de5927 WatchSource:0}: Error finding container fbfbad33f192567fb13c3ffca18e0ea060be189375e09c8bdef753ded3de5927: Status 404 returned error can't find the container with id fbfbad33f192567fb13c3ffca18e0ea060be189375e09c8bdef753ded3de5927 Sep 08 18:45:57.312178 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:57.312156 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd"] Sep 08 18:45:57.342223 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:45:57.342196 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fcd8c72_a05b_4b74_a88a_ff874fa717b6.slice/crio-6797cdbb2cca0a1ac43cc1ba65af297f20afcd13cb007842007073a8419ec8e0 WatchSource:0}: Error finding container 6797cdbb2cca0a1ac43cc1ba65af297f20afcd13cb007842007073a8419ec8e0: Status 404 returned error can't find the container with id 6797cdbb2cca0a1ac43cc1ba65af297f20afcd13cb007842007073a8419ec8e0 Sep 08 18:45:57.347343 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:57.347299 2553 generic.go:356] "Generic (PLEG): container finished" podID="035d85c9-1c2d-49f7-89b1-7224285292a3" containerID="78b527ac22b5181868e95d307f54db623955de999c0c3bd395c4ee6d2c52879e" exitCode=0 Sep 08 18:45:57.347458 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:57.347381 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" event={"ID":"035d85c9-1c2d-49f7-89b1-7224285292a3","Type":"ContainerDied","Data":"78b527ac22b5181868e95d307f54db623955de999c0c3bd395c4ee6d2c52879e"} Sep 08 18:45:57.347458 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:57.347423 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" event={"ID":"035d85c9-1c2d-49f7-89b1-7224285292a3","Type":"ContainerStarted","Data":"fbfbad33f192567fb13c3ffca18e0ea060be189375e09c8bdef753ded3de5927"} Sep 08 18:45:57.348413 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:57.348393 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 18:45:57.349108 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:57.349085 2553 generic.go:356] "Generic (PLEG): container finished" podID="f2b999ed-8ab7-46e0-8c1d-1f465e5666ee" containerID="6df3e69ec2b2873d8f45ef43856a2cb03a5ba30860c42a67114528873403fc65" exitCode=0 Sep 08 18:45:57.349203 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:57.349167 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" event={"ID":"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee","Type":"ContainerDied","Data":"6df3e69ec2b2873d8f45ef43856a2cb03a5ba30860c42a67114528873403fc65"} Sep 08 18:45:57.349267 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:57.349203 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" event={"ID":"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee","Type":"ContainerStarted","Data":"671e7a0598f7b750c0bc6e90e2049ef45458d0c5c73e6c91da681619e801b699"} Sep 08 18:45:57.350820 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:57.350798 2553 generic.go:356] "Generic (PLEG): container finished" podID="cb014585-9880-469b-9ed8-a22971997cca" containerID="90c83fbecde0af662fec13116aeb8c56584488d31cc26558181c97b0433157f4" exitCode=0 Sep 08 18:45:57.350914 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:57.350893 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" event={"ID":"cb014585-9880-469b-9ed8-a22971997cca","Type":"ContainerDied","Data":"90c83fbecde0af662fec13116aeb8c56584488d31cc26558181c97b0433157f4"} Sep 08 18:45:57.350958 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:57.350940 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" event={"ID":"cb014585-9880-469b-9ed8-a22971997cca","Type":"ContainerStarted","Data":"ffb55e1b093d1bc6bea5dc8d5501dc39861cc6d92a00f7f0189f1ea8adde4d88"} Sep 08 18:45:58.357912 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:58.357835 2553 generic.go:356] "Generic (PLEG): container finished" podID="3fcd8c72-a05b-4b74-a88a-ff874fa717b6" containerID="168be905a792903e6550795fd06bbd800317255178eceed2a37835fd01b58e73" exitCode=0 Sep 08 18:45:58.358319 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:58.357927 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" event={"ID":"3fcd8c72-a05b-4b74-a88a-ff874fa717b6","Type":"ContainerDied","Data":"168be905a792903e6550795fd06bbd800317255178eceed2a37835fd01b58e73"} Sep 08 18:45:58.358319 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:58.357983 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" event={"ID":"3fcd8c72-a05b-4b74-a88a-ff874fa717b6","Type":"ContainerStarted","Data":"6797cdbb2cca0a1ac43cc1ba65af297f20afcd13cb007842007073a8419ec8e0"} Sep 08 18:45:58.359866 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:58.359838 2553 generic.go:356] "Generic (PLEG): container finished" podID="f2b999ed-8ab7-46e0-8c1d-1f465e5666ee" containerID="78045cfe89a72c807f29d9efe548cb76220b6bd68e04fc5f9b9141e487265de6" exitCode=0 Sep 08 18:45:58.359984 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:58.359929 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" event={"ID":"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee","Type":"ContainerDied","Data":"78045cfe89a72c807f29d9efe548cb76220b6bd68e04fc5f9b9141e487265de6"} Sep 08 18:45:58.362090 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:58.362068 2553 generic.go:356] "Generic (PLEG): container finished" podID="cb014585-9880-469b-9ed8-a22971997cca" containerID="03b2b63c09ff3b8bdb14a053d43887137b1ba81e0cc4e1ad97adad3235ba23d2" exitCode=0 Sep 08 18:45:58.362173 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:58.362098 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" event={"ID":"cb014585-9880-469b-9ed8-a22971997cca","Type":"ContainerDied","Data":"03b2b63c09ff3b8bdb14a053d43887137b1ba81e0cc4e1ad97adad3235ba23d2"} Sep 08 18:45:59.368452 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:59.368358 2553 generic.go:356] "Generic (PLEG): container finished" podID="3fcd8c72-a05b-4b74-a88a-ff874fa717b6" containerID="c07bb5a1e3f7820de8da472d18aae655c586b42739a981dad65b0329e6e0ce36" exitCode=0 Sep 08 18:45:59.368791 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:59.368460 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" event={"ID":"3fcd8c72-a05b-4b74-a88a-ff874fa717b6","Type":"ContainerDied","Data":"c07bb5a1e3f7820de8da472d18aae655c586b42739a981dad65b0329e6e0ce36"} Sep 08 18:45:59.370426 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:59.370407 2553 generic.go:356] "Generic (PLEG): container finished" podID="035d85c9-1c2d-49f7-89b1-7224285292a3" containerID="ef42f8497d0840c8119c23ba9ad67c7c7031a6eea2e2791a031db37abadb4532" exitCode=0 Sep 08 18:45:59.370528 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:59.370453 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" event={"ID":"035d85c9-1c2d-49f7-89b1-7224285292a3","Type":"ContainerDied","Data":"ef42f8497d0840c8119c23ba9ad67c7c7031a6eea2e2791a031db37abadb4532"} Sep 08 18:45:59.372909 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:59.372889 2553 generic.go:356] "Generic (PLEG): container finished" podID="f2b999ed-8ab7-46e0-8c1d-1f465e5666ee" containerID="d6b875f46779ac14cfb23f67901f151ad458fba359f7cf384955010ab0d0b38a" exitCode=0 Sep 08 18:45:59.372980 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:59.372963 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" event={"ID":"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee","Type":"ContainerDied","Data":"d6b875f46779ac14cfb23f67901f151ad458fba359f7cf384955010ab0d0b38a"} Sep 08 18:45:59.375169 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:59.375148 2553 generic.go:356] "Generic (PLEG): container finished" podID="cb014585-9880-469b-9ed8-a22971997cca" containerID="6dbbc2072c37255943a9ce15f6f51c6cfa970efdefacb1a621e64f41e3fdaeed" exitCode=0 Sep 08 18:45:59.375258 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:45:59.375231 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" event={"ID":"cb014585-9880-469b-9ed8-a22971997cca","Type":"ContainerDied","Data":"6dbbc2072c37255943a9ce15f6f51c6cfa970efdefacb1a621e64f41e3fdaeed"} Sep 08 18:46:00.382279 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.382241 2553 generic.go:356] "Generic (PLEG): container finished" podID="035d85c9-1c2d-49f7-89b1-7224285292a3" containerID="f971ab24c5d4fea6011129587091b418a9b127585fa26e3775a81580a8924905" exitCode=0 Sep 08 18:46:00.382749 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.382321 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" event={"ID":"035d85c9-1c2d-49f7-89b1-7224285292a3","Type":"ContainerDied","Data":"f971ab24c5d4fea6011129587091b418a9b127585fa26e3775a81580a8924905"} Sep 08 18:46:00.384584 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.384560 2553 generic.go:356] "Generic (PLEG): container finished" podID="3fcd8c72-a05b-4b74-a88a-ff874fa717b6" containerID="4416fde2d9da447a00dc7ed27b34ded7e256ee68a417f8c7493acbb74f3f4c68" exitCode=0 Sep 08 18:46:00.384684 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.384625 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" event={"ID":"3fcd8c72-a05b-4b74-a88a-ff874fa717b6","Type":"ContainerDied","Data":"4416fde2d9da447a00dc7ed27b34ded7e256ee68a417f8c7493acbb74f3f4c68"} Sep 08 18:46:00.543357 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.543334 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" Sep 08 18:46:00.546121 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.546104 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" Sep 08 18:46:00.630192 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.630162 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb014585-9880-469b-9ed8-a22971997cca-bundle\") pod \"cb014585-9880-469b-9ed8-a22971997cca\" (UID: \"cb014585-9880-469b-9ed8-a22971997cca\") " Sep 08 18:46:00.630192 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.630195 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmpdv\" (UniqueName: \"kubernetes.io/projected/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-kube-api-access-gmpdv\") pod \"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee\" (UID: \"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee\") " Sep 08 18:46:00.630464 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.630211 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-util\") pod \"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee\" (UID: \"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee\") " Sep 08 18:46:00.630464 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.630233 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f69w4\" (UniqueName: \"kubernetes.io/projected/cb014585-9880-469b-9ed8-a22971997cca-kube-api-access-f69w4\") pod \"cb014585-9880-469b-9ed8-a22971997cca\" (UID: \"cb014585-9880-469b-9ed8-a22971997cca\") " Sep 08 18:46:00.630464 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.630264 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-bundle\") pod \"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee\" (UID: \"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee\") " Sep 08 18:46:00.630464 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.630384 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb014585-9880-469b-9ed8-a22971997cca-util\") pod \"cb014585-9880-469b-9ed8-a22971997cca\" (UID: \"cb014585-9880-469b-9ed8-a22971997cca\") " Sep 08 18:46:00.630881 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.630857 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb014585-9880-469b-9ed8-a22971997cca-bundle" (OuterVolumeSpecName: "bundle") pod "cb014585-9880-469b-9ed8-a22971997cca" (UID: "cb014585-9880-469b-9ed8-a22971997cca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:46:00.631264 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.631232 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-bundle" (OuterVolumeSpecName: "bundle") pod "f2b999ed-8ab7-46e0-8c1d-1f465e5666ee" (UID: "f2b999ed-8ab7-46e0-8c1d-1f465e5666ee"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:46:00.632352 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.632296 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb014585-9880-469b-9ed8-a22971997cca-kube-api-access-f69w4" (OuterVolumeSpecName: "kube-api-access-f69w4") pod "cb014585-9880-469b-9ed8-a22971997cca" (UID: "cb014585-9880-469b-9ed8-a22971997cca"). InnerVolumeSpecName "kube-api-access-f69w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:46:00.632642 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.632623 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-kube-api-access-gmpdv" (OuterVolumeSpecName: "kube-api-access-gmpdv") pod "f2b999ed-8ab7-46e0-8c1d-1f465e5666ee" (UID: "f2b999ed-8ab7-46e0-8c1d-1f465e5666ee"). InnerVolumeSpecName "kube-api-access-gmpdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:46:00.635847 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.635817 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-util" (OuterVolumeSpecName: "util") pod "f2b999ed-8ab7-46e0-8c1d-1f465e5666ee" (UID: "f2b999ed-8ab7-46e0-8c1d-1f465e5666ee"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:46:00.637022 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.636997 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb014585-9880-469b-9ed8-a22971997cca-util" (OuterVolumeSpecName: "util") pod "cb014585-9880-469b-9ed8-a22971997cca" (UID: "cb014585-9880-469b-9ed8-a22971997cca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:46:00.731575 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.731548 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb014585-9880-469b-9ed8-a22971997cca-util\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:46:00.731575 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.731572 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb014585-9880-469b-9ed8-a22971997cca-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:46:00.731721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.731581 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmpdv\" (UniqueName: \"kubernetes.io/projected/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-kube-api-access-gmpdv\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:46:00.731721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.731591 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-util\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:46:00.731721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.731599 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f69w4\" (UniqueName: \"kubernetes.io/projected/cb014585-9880-469b-9ed8-a22971997cca-kube-api-access-f69w4\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:46:00.731721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:00.731607 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2b999ed-8ab7-46e0-8c1d-1f465e5666ee-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:46:01.391770 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.391734 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" event={"ID":"f2b999ed-8ab7-46e0-8c1d-1f465e5666ee","Type":"ContainerDied","Data":"671e7a0598f7b750c0bc6e90e2049ef45458d0c5c73e6c91da681619e801b699"} Sep 08 18:46:01.392149 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.391791 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="671e7a0598f7b750c0bc6e90e2049ef45458d0c5c73e6c91da681619e801b699" Sep 08 18:46:01.392149 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.391748 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d799hbr" Sep 08 18:46:01.393899 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.393853 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" Sep 08 18:46:01.393899 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.393859 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4csws8c" event={"ID":"cb014585-9880-469b-9ed8-a22971997cca","Type":"ContainerDied","Data":"ffb55e1b093d1bc6bea5dc8d5501dc39861cc6d92a00f7f0189f1ea8adde4d88"} Sep 08 18:46:01.394069 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.393931 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffb55e1b093d1bc6bea5dc8d5501dc39861cc6d92a00f7f0189f1ea8adde4d88" Sep 08 18:46:01.542621 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.542599 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" Sep 08 18:46:01.544479 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.544460 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" Sep 08 18:46:01.639641 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.639606 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5fvj\" (UniqueName: \"kubernetes.io/projected/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-kube-api-access-h5fvj\") pod \"3fcd8c72-a05b-4b74-a88a-ff874fa717b6\" (UID: \"3fcd8c72-a05b-4b74-a88a-ff874fa717b6\") " Sep 08 18:46:01.639641 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.639645 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-util\") pod \"3fcd8c72-a05b-4b74-a88a-ff874fa717b6\" (UID: \"3fcd8c72-a05b-4b74-a88a-ff874fa717b6\") " Sep 08 18:46:01.639855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.639673 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rt47\" (UniqueName: \"kubernetes.io/projected/035d85c9-1c2d-49f7-89b1-7224285292a3-kube-api-access-4rt47\") pod \"035d85c9-1c2d-49f7-89b1-7224285292a3\" (UID: \"035d85c9-1c2d-49f7-89b1-7224285292a3\") " Sep 08 18:46:01.639855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.639696 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/035d85c9-1c2d-49f7-89b1-7224285292a3-bundle\") pod \"035d85c9-1c2d-49f7-89b1-7224285292a3\" (UID: \"035d85c9-1c2d-49f7-89b1-7224285292a3\") " Sep 08 18:46:01.639855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.639729 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-bundle\") pod \"3fcd8c72-a05b-4b74-a88a-ff874fa717b6\" (UID: \"3fcd8c72-a05b-4b74-a88a-ff874fa717b6\") " Sep 08 18:46:01.639855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.639799 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/035d85c9-1c2d-49f7-89b1-7224285292a3-util\") pod \"035d85c9-1c2d-49f7-89b1-7224285292a3\" (UID: \"035d85c9-1c2d-49f7-89b1-7224285292a3\") " Sep 08 18:46:01.640297 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.640191 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035d85c9-1c2d-49f7-89b1-7224285292a3-bundle" (OuterVolumeSpecName: "bundle") pod "035d85c9-1c2d-49f7-89b1-7224285292a3" (UID: "035d85c9-1c2d-49f7-89b1-7224285292a3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:46:01.640593 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.640568 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-bundle" (OuterVolumeSpecName: "bundle") pod "3fcd8c72-a05b-4b74-a88a-ff874fa717b6" (UID: "3fcd8c72-a05b-4b74-a88a-ff874fa717b6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:46:01.641824 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.641774 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/035d85c9-1c2d-49f7-89b1-7224285292a3-kube-api-access-4rt47" (OuterVolumeSpecName: "kube-api-access-4rt47") pod "035d85c9-1c2d-49f7-89b1-7224285292a3" (UID: "035d85c9-1c2d-49f7-89b1-7224285292a3"). InnerVolumeSpecName "kube-api-access-4rt47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:46:01.641970 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.641947 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-kube-api-access-h5fvj" (OuterVolumeSpecName: "kube-api-access-h5fvj") pod "3fcd8c72-a05b-4b74-a88a-ff874fa717b6" (UID: "3fcd8c72-a05b-4b74-a88a-ff874fa717b6"). InnerVolumeSpecName "kube-api-access-h5fvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:46:01.645477 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.645432 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035d85c9-1c2d-49f7-89b1-7224285292a3-util" (OuterVolumeSpecName: "util") pod "035d85c9-1c2d-49f7-89b1-7224285292a3" (UID: "035d85c9-1c2d-49f7-89b1-7224285292a3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:46:01.645566 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.645551 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-util" (OuterVolumeSpecName: "util") pod "3fcd8c72-a05b-4b74-a88a-ff874fa717b6" (UID: "3fcd8c72-a05b-4b74-a88a-ff874fa717b6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:46:01.740632 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.740594 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5fvj\" (UniqueName: \"kubernetes.io/projected/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-kube-api-access-h5fvj\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:46:01.740632 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.740627 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-util\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:46:01.740843 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.740643 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rt47\" (UniqueName: \"kubernetes.io/projected/035d85c9-1c2d-49f7-89b1-7224285292a3-kube-api-access-4rt47\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:46:01.740843 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.740656 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/035d85c9-1c2d-49f7-89b1-7224285292a3-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:46:01.740843 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.740667 2553 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3fcd8c72-a05b-4b74-a88a-ff874fa717b6-bundle\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:46:01.740843 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:01.740677 2553 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/035d85c9-1c2d-49f7-89b1-7224285292a3-util\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:46:02.400627 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:02.400597 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" Sep 08 18:46:02.400627 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:02.400616 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azsnsd" event={"ID":"3fcd8c72-a05b-4b74-a88a-ff874fa717b6","Type":"ContainerDied","Data":"6797cdbb2cca0a1ac43cc1ba65af297f20afcd13cb007842007073a8419ec8e0"} Sep 08 18:46:02.401117 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:02.400655 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6797cdbb2cca0a1ac43cc1ba65af297f20afcd13cb007842007073a8419ec8e0" Sep 08 18:46:02.402582 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:02.402556 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" event={"ID":"035d85c9-1c2d-49f7-89b1-7224285292a3","Type":"ContainerDied","Data":"fbfbad33f192567fb13c3ffca18e0ea060be189375e09c8bdef753ded3de5927"} Sep 08 18:46:02.402693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:02.402602 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbfbad33f192567fb13c3ffca18e0ea060be189375e09c8bdef753ded3de5927" Sep 08 18:46:02.402693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:46:02.402565 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mz257" Sep 08 18:47:56.089141 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089106 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6fb4457bfb-hq8mg"] Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089423 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="035d85c9-1c2d-49f7-89b1-7224285292a3" containerName="util" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089434 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="035d85c9-1c2d-49f7-89b1-7224285292a3" containerName="util" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089441 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b999ed-8ab7-46e0-8c1d-1f465e5666ee" containerName="pull" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089447 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b999ed-8ab7-46e0-8c1d-1f465e5666ee" containerName="pull" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089454 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb014585-9880-469b-9ed8-a22971997cca" containerName="extract" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089460 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb014585-9880-469b-9ed8-a22971997cca" containerName="extract" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089468 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fcd8c72-a05b-4b74-a88a-ff874fa717b6" containerName="pull" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089473 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fcd8c72-a05b-4b74-a88a-ff874fa717b6" containerName="pull" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089485 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb014585-9880-469b-9ed8-a22971997cca" containerName="util" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089490 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb014585-9880-469b-9ed8-a22971997cca" containerName="util" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089500 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b999ed-8ab7-46e0-8c1d-1f465e5666ee" containerName="util" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089505 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b999ed-8ab7-46e0-8c1d-1f465e5666ee" containerName="util" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089511 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b999ed-8ab7-46e0-8c1d-1f465e5666ee" containerName="extract" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089517 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b999ed-8ab7-46e0-8c1d-1f465e5666ee" containerName="extract" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089522 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="035d85c9-1c2d-49f7-89b1-7224285292a3" containerName="pull" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089527 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="035d85c9-1c2d-49f7-89b1-7224285292a3" containerName="pull" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089535 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb014585-9880-469b-9ed8-a22971997cca" containerName="pull" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089540 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb014585-9880-469b-9ed8-a22971997cca" containerName="pull" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089544 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fcd8c72-a05b-4b74-a88a-ff874fa717b6" containerName="extract" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089549 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fcd8c72-a05b-4b74-a88a-ff874fa717b6" containerName="extract" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089555 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="035d85c9-1c2d-49f7-89b1-7224285292a3" containerName="extract" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089560 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="035d85c9-1c2d-49f7-89b1-7224285292a3" containerName="extract" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089571 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fcd8c72-a05b-4b74-a88a-ff874fa717b6" containerName="util" Sep 08 18:47:56.089585 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089576 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fcd8c72-a05b-4b74-a88a-ff874fa717b6" containerName="util" Sep 08 18:47:56.090272 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089625 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="035d85c9-1c2d-49f7-89b1-7224285292a3" containerName="extract" Sep 08 18:47:56.090272 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089633 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb014585-9880-469b-9ed8-a22971997cca" containerName="extract" Sep 08 18:47:56.090272 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089641 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b999ed-8ab7-46e0-8c1d-1f465e5666ee" containerName="extract" Sep 08 18:47:56.090272 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.089648 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fcd8c72-a05b-4b74-a88a-ff874fa717b6" containerName="extract" Sep 08 18:47:56.091393 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.091353 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" Sep 08 18:47:56.093955 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.093931 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 18:47:56.094130 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.094103 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 18:47:56.094241 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.094134 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pxxwb\"" Sep 08 18:47:56.095035 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.095020 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 18:47:56.104960 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.104937 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fb4457bfb-hq8mg"] Sep 08 18:47:56.176209 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.176180 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df5637e2-7d48-4d1f-8567-084d02d3420d-metrics-certs\") pod \"maas-controller-6fb4457bfb-hq8mg\" (UID: \"df5637e2-7d48-4d1f-8567-084d02d3420d\") " pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" Sep 08 18:47:56.176389 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.176236 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57xwq\" (UniqueName: \"kubernetes.io/projected/df5637e2-7d48-4d1f-8567-084d02d3420d-kube-api-access-57xwq\") pod \"maas-controller-6fb4457bfb-hq8mg\" (UID: \"df5637e2-7d48-4d1f-8567-084d02d3420d\") " pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" Sep 08 18:47:56.176389 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.176286 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df5637e2-7d48-4d1f-8567-084d02d3420d-webhook-cert\") pod \"maas-controller-6fb4457bfb-hq8mg\" (UID: \"df5637e2-7d48-4d1f-8567-084d02d3420d\") " pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" Sep 08 18:47:56.264860 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.264821 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68c8c7d87-vxscg"] Sep 08 18:47:56.267350 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.267328 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8c7d87-vxscg" Sep 08 18:47:56.277141 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.277116 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df5637e2-7d48-4d1f-8567-084d02d3420d-metrics-certs\") pod \"maas-controller-6fb4457bfb-hq8mg\" (UID: \"df5637e2-7d48-4d1f-8567-084d02d3420d\") " pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" Sep 08 18:47:56.277277 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.277179 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57xwq\" (UniqueName: \"kubernetes.io/projected/df5637e2-7d48-4d1f-8567-084d02d3420d-kube-api-access-57xwq\") pod \"maas-controller-6fb4457bfb-hq8mg\" (UID: \"df5637e2-7d48-4d1f-8567-084d02d3420d\") " pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" Sep 08 18:47:56.277277 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.277212 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df5637e2-7d48-4d1f-8567-084d02d3420d-webhook-cert\") pod \"maas-controller-6fb4457bfb-hq8mg\" (UID: \"df5637e2-7d48-4d1f-8567-084d02d3420d\") " pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" Sep 08 18:47:56.277642 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.277620 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c8c7d87-vxscg"] Sep 08 18:47:56.279798 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.279767 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df5637e2-7d48-4d1f-8567-084d02d3420d-metrics-certs\") pod \"maas-controller-6fb4457bfb-hq8mg\" (UID: \"df5637e2-7d48-4d1f-8567-084d02d3420d\") " pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" Sep 08 18:47:56.279976 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.279950 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df5637e2-7d48-4d1f-8567-084d02d3420d-webhook-cert\") pod \"maas-controller-6fb4457bfb-hq8mg\" (UID: \"df5637e2-7d48-4d1f-8567-084d02d3420d\") " pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" Sep 08 18:47:56.285094 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.285069 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57xwq\" (UniqueName: \"kubernetes.io/projected/df5637e2-7d48-4d1f-8567-084d02d3420d-kube-api-access-57xwq\") pod \"maas-controller-6fb4457bfb-hq8mg\" (UID: \"df5637e2-7d48-4d1f-8567-084d02d3420d\") " pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" Sep 08 18:47:56.378203 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.378177 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3767535-1475-4933-8594-3e2a79ce4b2b-metrics-certs\") pod \"maas-controller-68c8c7d87-vxscg\" (UID: \"e3767535-1475-4933-8594-3e2a79ce4b2b\") " pod="opendatahub/maas-controller-68c8c7d87-vxscg" Sep 08 18:47:56.378346 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.378240 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kfk5\" (UniqueName: \"kubernetes.io/projected/e3767535-1475-4933-8594-3e2a79ce4b2b-kube-api-access-8kfk5\") pod \"maas-controller-68c8c7d87-vxscg\" (UID: \"e3767535-1475-4933-8594-3e2a79ce4b2b\") " pod="opendatahub/maas-controller-68c8c7d87-vxscg" Sep 08 18:47:56.378346 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.378289 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3767535-1475-4933-8594-3e2a79ce4b2b-webhook-cert\") pod \"maas-controller-68c8c7d87-vxscg\" (UID: \"e3767535-1475-4933-8594-3e2a79ce4b2b\") " pod="opendatahub/maas-controller-68c8c7d87-vxscg" Sep 08 18:47:56.401143 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.401120 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" Sep 08 18:47:56.479069 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.479039 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3767535-1475-4933-8594-3e2a79ce4b2b-webhook-cert\") pod \"maas-controller-68c8c7d87-vxscg\" (UID: \"e3767535-1475-4933-8594-3e2a79ce4b2b\") " pod="opendatahub/maas-controller-68c8c7d87-vxscg" Sep 08 18:47:56.479240 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.479084 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3767535-1475-4933-8594-3e2a79ce4b2b-metrics-certs\") pod \"maas-controller-68c8c7d87-vxscg\" (UID: \"e3767535-1475-4933-8594-3e2a79ce4b2b\") " pod="opendatahub/maas-controller-68c8c7d87-vxscg" Sep 08 18:47:56.479240 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.479138 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kfk5\" (UniqueName: \"kubernetes.io/projected/e3767535-1475-4933-8594-3e2a79ce4b2b-kube-api-access-8kfk5\") pod \"maas-controller-68c8c7d87-vxscg\" (UID: \"e3767535-1475-4933-8594-3e2a79ce4b2b\") " pod="opendatahub/maas-controller-68c8c7d87-vxscg" Sep 08 18:47:56.481501 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.481477 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3767535-1475-4933-8594-3e2a79ce4b2b-metrics-certs\") pod \"maas-controller-68c8c7d87-vxscg\" (UID: \"e3767535-1475-4933-8594-3e2a79ce4b2b\") " pod="opendatahub/maas-controller-68c8c7d87-vxscg" Sep 08 18:47:56.481628 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.481511 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3767535-1475-4933-8594-3e2a79ce4b2b-webhook-cert\") pod \"maas-controller-68c8c7d87-vxscg\" (UID: \"e3767535-1475-4933-8594-3e2a79ce4b2b\") " pod="opendatahub/maas-controller-68c8c7d87-vxscg" Sep 08 18:47:56.487448 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.487424 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kfk5\" (UniqueName: \"kubernetes.io/projected/e3767535-1475-4933-8594-3e2a79ce4b2b-kube-api-access-8kfk5\") pod \"maas-controller-68c8c7d87-vxscg\" (UID: \"e3767535-1475-4933-8594-3e2a79ce4b2b\") " pod="opendatahub/maas-controller-68c8c7d87-vxscg" Sep 08 18:47:56.577304 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.577274 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8c7d87-vxscg" Sep 08 18:47:56.669248 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.669223 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fb4457bfb-hq8mg"] Sep 08 18:47:56.670460 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:47:56.670434 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5637e2_7d48_4d1f_8567_084d02d3420d.slice/crio-41aca263ceaa789ac59f6df41c53e3f2ccbc096be61783695f2b2a806f65c334 WatchSource:0}: Error finding container 41aca263ceaa789ac59f6df41c53e3f2ccbc096be61783695f2b2a806f65c334: Status 404 returned error can't find the container with id 41aca263ceaa789ac59f6df41c53e3f2ccbc096be61783695f2b2a806f65c334 Sep 08 18:47:56.851387 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.851349 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c8c7d87-vxscg"] Sep 08 18:47:56.852615 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:47:56.852588 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3767535_1475_4933_8594_3e2a79ce4b2b.slice/crio-ee23ec0e0d94aab8130388b7b4383491989313ae0c150df9444981c26f7cc43f WatchSource:0}: Error finding container ee23ec0e0d94aab8130388b7b4383491989313ae0c150df9444981c26f7cc43f: Status 404 returned error can't find the container with id ee23ec0e0d94aab8130388b7b4383491989313ae0c150df9444981c26f7cc43f Sep 08 18:47:56.948463 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.948387 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8c7d87-vxscg" event={"ID":"e3767535-1475-4933-8594-3e2a79ce4b2b","Type":"ContainerStarted","Data":"ee23ec0e0d94aab8130388b7b4383491989313ae0c150df9444981c26f7cc43f"} Sep 08 18:47:56.949530 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:56.949505 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" event={"ID":"df5637e2-7d48-4d1f-8567-084d02d3420d","Type":"ContainerStarted","Data":"41aca263ceaa789ac59f6df41c53e3f2ccbc096be61783695f2b2a806f65c334"} Sep 08 18:47:59.972592 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:59.972549 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" event={"ID":"df5637e2-7d48-4d1f-8567-084d02d3420d","Type":"ContainerStarted","Data":"c64efc2bc31d8283aaf229145b33e0ecd747d53317031675b7b8ebd9d1e571d5"} Sep 08 18:47:59.973021 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:59.972710 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" Sep 08 18:47:59.974142 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:59.974120 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8c7d87-vxscg" event={"ID":"e3767535-1475-4933-8594-3e2a79ce4b2b","Type":"ContainerStarted","Data":"c213b765b23cf29acb7406653a945ed55d4ca2fe2a9d197af2108e487ebad2e5"} Sep 08 18:47:59.974252 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:47:59.974170 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68c8c7d87-vxscg" Sep 08 18:48:00.007292 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:00.007242 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" podStartSLOduration=1.477202009 podStartE2EDuration="4.007228809s" totalImagesPullingTime="2.5300268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:48:00.005801362 +0000 UTC m=+815.711689594" watchObservedRunningTime="2026-09-08 18:48:00.007228809 +0000 UTC m=+815.713117041" Sep 08 18:48:00.031450 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:00.031396 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68c8c7d87-vxscg" podStartSLOduration=1.6741980239999998 podStartE2EDuration="4.031380229s" totalImagesPullingTime="2.357182205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:48:00.029059403 +0000 UTC m=+815.734947654" watchObservedRunningTime="2026-09-08 18:48:00.031380229 +0000 UTC m=+815.737268457" Sep 08 18:48:10.981243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:10.981211 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" Sep 08 18:48:10.981732 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:10.981331 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68c8c7d87-vxscg" Sep 08 18:48:11.046039 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:11.046007 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fb4457bfb-hq8mg"] Sep 08 18:48:11.046529 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:11.046351 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" podUID="df5637e2-7d48-4d1f-8567-084d02d3420d" containerName="manager" containerID="cri-o://c64efc2bc31d8283aaf229145b33e0ecd747d53317031675b7b8ebd9d1e571d5" gracePeriod=30 Sep 08 18:48:11.287500 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:11.287478 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" Sep 08 18:48:11.403127 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:11.403097 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df5637e2-7d48-4d1f-8567-084d02d3420d-metrics-certs\") pod \"df5637e2-7d48-4d1f-8567-084d02d3420d\" (UID: \"df5637e2-7d48-4d1f-8567-084d02d3420d\") " Sep 08 18:48:11.403297 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:11.403173 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57xwq\" (UniqueName: \"kubernetes.io/projected/df5637e2-7d48-4d1f-8567-084d02d3420d-kube-api-access-57xwq\") pod \"df5637e2-7d48-4d1f-8567-084d02d3420d\" (UID: \"df5637e2-7d48-4d1f-8567-084d02d3420d\") " Sep 08 18:48:11.403297 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:11.403265 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df5637e2-7d48-4d1f-8567-084d02d3420d-webhook-cert\") pod \"df5637e2-7d48-4d1f-8567-084d02d3420d\" (UID: \"df5637e2-7d48-4d1f-8567-084d02d3420d\") " Sep 08 18:48:11.405400 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:11.405373 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5637e2-7d48-4d1f-8567-084d02d3420d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "df5637e2-7d48-4d1f-8567-084d02d3420d" (UID: "df5637e2-7d48-4d1f-8567-084d02d3420d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:48:11.405494 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:11.405400 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5637e2-7d48-4d1f-8567-084d02d3420d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "df5637e2-7d48-4d1f-8567-084d02d3420d" (UID: "df5637e2-7d48-4d1f-8567-084d02d3420d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 18:48:11.405494 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:11.405484 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5637e2-7d48-4d1f-8567-084d02d3420d-kube-api-access-57xwq" (OuterVolumeSpecName: "kube-api-access-57xwq") pod "df5637e2-7d48-4d1f-8567-084d02d3420d" (UID: "df5637e2-7d48-4d1f-8567-084d02d3420d"). InnerVolumeSpecName "kube-api-access-57xwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:48:11.504876 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:11.504849 2553 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df5637e2-7d48-4d1f-8567-084d02d3420d-webhook-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:48:11.504876 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:11.504872 2553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df5637e2-7d48-4d1f-8567-084d02d3420d-metrics-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:48:11.505025 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:11.504883 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57xwq\" (UniqueName: \"kubernetes.io/projected/df5637e2-7d48-4d1f-8567-084d02d3420d-kube-api-access-57xwq\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:48:12.034210 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:12.034172 2553 generic.go:356] "Generic (PLEG): container finished" podID="df5637e2-7d48-4d1f-8567-084d02d3420d" containerID="c64efc2bc31d8283aaf229145b33e0ecd747d53317031675b7b8ebd9d1e571d5" exitCode=0 Sep 08 18:48:12.034675 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:12.034230 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" Sep 08 18:48:12.034675 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:12.034261 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" event={"ID":"df5637e2-7d48-4d1f-8567-084d02d3420d","Type":"ContainerDied","Data":"c64efc2bc31d8283aaf229145b33e0ecd747d53317031675b7b8ebd9d1e571d5"} Sep 08 18:48:12.034675 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:12.034318 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fb4457bfb-hq8mg" event={"ID":"df5637e2-7d48-4d1f-8567-084d02d3420d","Type":"ContainerDied","Data":"41aca263ceaa789ac59f6df41c53e3f2ccbc096be61783695f2b2a806f65c334"} Sep 08 18:48:12.034675 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:12.034341 2553 scope.go:117] "RemoveContainer" containerID="c64efc2bc31d8283aaf229145b33e0ecd747d53317031675b7b8ebd9d1e571d5" Sep 08 18:48:12.043653 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:12.043634 2553 scope.go:117] "RemoveContainer" containerID="c64efc2bc31d8283aaf229145b33e0ecd747d53317031675b7b8ebd9d1e571d5" Sep 08 18:48:12.043922 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:48:12.043903 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64efc2bc31d8283aaf229145b33e0ecd747d53317031675b7b8ebd9d1e571d5\": container with ID starting with c64efc2bc31d8283aaf229145b33e0ecd747d53317031675b7b8ebd9d1e571d5 not found: ID does not exist" containerID="c64efc2bc31d8283aaf229145b33e0ecd747d53317031675b7b8ebd9d1e571d5" Sep 08 18:48:12.043968 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:12.043930 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64efc2bc31d8283aaf229145b33e0ecd747d53317031675b7b8ebd9d1e571d5"} err="failed to get container status \"c64efc2bc31d8283aaf229145b33e0ecd747d53317031675b7b8ebd9d1e571d5\": rpc error: code = NotFound desc = could not find container \"c64efc2bc31d8283aaf229145b33e0ecd747d53317031675b7b8ebd9d1e571d5\": container with ID starting with c64efc2bc31d8283aaf229145b33e0ecd747d53317031675b7b8ebd9d1e571d5 not found: ID does not exist" Sep 08 18:48:12.056232 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:12.056212 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fb4457bfb-hq8mg"] Sep 08 18:48:12.062330 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:12.062309 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6fb4457bfb-hq8mg"] Sep 08 18:48:12.745504 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:12.745474 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df5637e2-7d48-4d1f-8567-084d02d3420d" path="/var/lib/kubelet/pods/df5637e2-7d48-4d1f-8567-084d02d3420d/volumes" Sep 08 18:48:32.881889 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:32.881854 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc"] Sep 08 18:48:32.882245 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:32.882157 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df5637e2-7d48-4d1f-8567-084d02d3420d" containerName="manager" Sep 08 18:48:32.882245 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:32.882168 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="df5637e2-7d48-4d1f-8567-084d02d3420d" containerName="manager" Sep 08 18:48:32.882245 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:32.882235 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="df5637e2-7d48-4d1f-8567-084d02d3420d" containerName="manager" Sep 08 18:48:32.890739 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:32.890715 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc" Sep 08 18:48:32.893657 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:32.893624 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 08 18:48:32.893803 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:32.893725 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-g5jdp\"" Sep 08 18:48:32.895039 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:32.895018 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc"] Sep 08 18:48:32.982711 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:32.982675 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvwzq\" (UniqueName: \"kubernetes.io/projected/43765cab-e58f-47fb-9b10-ac65bae9b8e3-kube-api-access-mvwzq\") pod \"payload-pre-processing-59fbc47bf8-s6mvc\" (UID: \"43765cab-e58f-47fb-9b10-ac65bae9b8e3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc" Sep 08 18:48:32.982839 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:32.982740 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43765cab-e58f-47fb-9b10-ac65bae9b8e3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-s6mvc\" (UID: \"43765cab-e58f-47fb-9b10-ac65bae9b8e3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc" Sep 08 18:48:33.083993 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:33.083957 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43765cab-e58f-47fb-9b10-ac65bae9b8e3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-s6mvc\" (UID: \"43765cab-e58f-47fb-9b10-ac65bae9b8e3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc" Sep 08 18:48:33.084164 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:33.084139 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvwzq\" (UniqueName: \"kubernetes.io/projected/43765cab-e58f-47fb-9b10-ac65bae9b8e3-kube-api-access-mvwzq\") pod \"payload-pre-processing-59fbc47bf8-s6mvc\" (UID: \"43765cab-e58f-47fb-9b10-ac65bae9b8e3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc" Sep 08 18:48:33.084678 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:33.084659 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43765cab-e58f-47fb-9b10-ac65bae9b8e3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-s6mvc\" (UID: \"43765cab-e58f-47fb-9b10-ac65bae9b8e3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc" Sep 08 18:48:33.095446 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:33.095417 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvwzq\" (UniqueName: \"kubernetes.io/projected/43765cab-e58f-47fb-9b10-ac65bae9b8e3-kube-api-access-mvwzq\") pod \"payload-pre-processing-59fbc47bf8-s6mvc\" (UID: \"43765cab-e58f-47fb-9b10-ac65bae9b8e3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc" Sep 08 18:48:33.202119 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:33.202032 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc" Sep 08 18:48:33.473798 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:33.473772 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc"] Sep 08 18:48:33.475176 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:48:33.475145 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43765cab_e58f_47fb_9b10_ac65bae9b8e3.slice/crio-3508b5b3af8ab6379adf4a937294b8ea861cce34318ff5ce2a701d1d037f1304 WatchSource:0}: Error finding container 3508b5b3af8ab6379adf4a937294b8ea861cce34318ff5ce2a701d1d037f1304: Status 404 returned error can't find the container with id 3508b5b3af8ab6379adf4a937294b8ea861cce34318ff5ce2a701d1d037f1304 Sep 08 18:48:34.149252 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:34.149123 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc" event={"ID":"43765cab-e58f-47fb-9b10-ac65bae9b8e3","Type":"ContainerStarted","Data":"3508b5b3af8ab6379adf4a937294b8ea861cce34318ff5ce2a701d1d037f1304"} Sep 08 18:48:36.167448 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:36.167407 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc" event={"ID":"43765cab-e58f-47fb-9b10-ac65bae9b8e3","Type":"ContainerStarted","Data":"254353aedc7bae7ea3a0e8fc8fc3f008f9fd1f16ae52f457e54126272880ca9b"} Sep 08 18:48:36.167811 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:36.167514 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc" Sep 08 18:48:36.184000 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:36.183954 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc" podStartSLOduration=2.4610866270000002 podStartE2EDuration="4.183940534s" totalImagesPullingTime="1.722853907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:48:36.181490199 +0000 UTC m=+851.887378431" watchObservedRunningTime="2026-09-08 18:48:36.183940534 +0000 UTC m=+851.889828765" Sep 08 18:48:47.173954 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:48:47.173920 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-s6mvc" Sep 08 18:49:23.318244 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.318166 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b"] Sep 08 18:49:23.320642 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.320618 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.327205 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.327181 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 08 18:49:23.327583 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.327565 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 08 18:49:23.327639 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.327583 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 08 18:49:23.327787 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.327767 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-htg2r\"" Sep 08 18:49:23.351278 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.351249 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b"] Sep 08 18:49:23.402699 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.402676 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d13b3600-44b7-490a-ba2f-6d96a80be4b5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.402792 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.402713 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d13b3600-44b7-490a-ba2f-6d96a80be4b5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.402792 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.402736 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr28w\" (UniqueName: \"kubernetes.io/projected/d13b3600-44b7-490a-ba2f-6d96a80be4b5-kube-api-access-rr28w\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.402877 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.402786 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d13b3600-44b7-490a-ba2f-6d96a80be4b5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.402877 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.402829 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d13b3600-44b7-490a-ba2f-6d96a80be4b5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.402877 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.402875 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d13b3600-44b7-490a-ba2f-6d96a80be4b5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.503703 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.503666 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d13b3600-44b7-490a-ba2f-6d96a80be4b5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.503880 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.503718 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d13b3600-44b7-490a-ba2f-6d96a80be4b5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.503880 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.503768 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d13b3600-44b7-490a-ba2f-6d96a80be4b5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.503880 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.503791 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d13b3600-44b7-490a-ba2f-6d96a80be4b5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.503880 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.503862 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr28w\" (UniqueName: \"kubernetes.io/projected/d13b3600-44b7-490a-ba2f-6d96a80be4b5-kube-api-access-rr28w\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.504113 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.503901 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d13b3600-44b7-490a-ba2f-6d96a80be4b5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.504193 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.504171 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d13b3600-44b7-490a-ba2f-6d96a80be4b5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.504275 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.504251 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d13b3600-44b7-490a-ba2f-6d96a80be4b5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.504331 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.504313 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d13b3600-44b7-490a-ba2f-6d96a80be4b5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.506108 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.506085 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d13b3600-44b7-490a-ba2f-6d96a80be4b5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.506296 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.506276 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d13b3600-44b7-490a-ba2f-6d96a80be4b5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.512597 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.512575 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr28w\" (UniqueName: \"kubernetes.io/projected/d13b3600-44b7-490a-ba2f-6d96a80be4b5-kube-api-access-rr28w\") pod \"e2e-distinct-simulated-kserve-59975d69f9-n7l5b\" (UID: \"d13b3600-44b7-490a-ba2f-6d96a80be4b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.647459 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.647427 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:23.915218 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:23.915192 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b"] Sep 08 18:49:23.916606 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:49:23.916583 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13b3600_44b7_490a_ba2f_6d96a80be4b5.slice/crio-9deee81266c588a48826539340c354dcab7aee928da113430eda62331ea82137 WatchSource:0}: Error finding container 9deee81266c588a48826539340c354dcab7aee928da113430eda62331ea82137: Status 404 returned error can't find the container with id 9deee81266c588a48826539340c354dcab7aee928da113430eda62331ea82137 Sep 08 18:49:24.398965 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:24.398929 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" event={"ID":"d13b3600-44b7-490a-ba2f-6d96a80be4b5","Type":"ContainerStarted","Data":"9deee81266c588a48826539340c354dcab7aee928da113430eda62331ea82137"} Sep 08 18:49:24.837043 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:24.836961 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 18:49:24.837206 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:24.837083 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 18:49:24.840974 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:24.840955 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 18:49:24.841110 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:24.841013 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 18:49:25.407262 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:25.407225 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" event={"ID":"d13b3600-44b7-490a-ba2f-6d96a80be4b5","Type":"ContainerStarted","Data":"9fde3d10465147cb2aa5553891f582757aef56b93b101c1b4b6d6cc6cf63ad45"} Sep 08 18:49:25.424819 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:25.424770 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" podStartSLOduration=1.499082996 podStartE2EDuration="2.424753617s" totalImagesPullingTime="925.670621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:49:25.422170221 +0000 UTC m=+901.128058457" watchObservedRunningTime="2026-09-08 18:49:25.424753617 +0000 UTC m=+901.130641850" Sep 08 18:49:33.570382 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.570335 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n"] Sep 08 18:49:33.572857 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.572837 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.575617 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.575594 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 08 18:49:33.583589 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.583567 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n"] Sep 08 18:49:33.597486 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.597465 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2670c956-bd01-4574-b847-3a1b5c0ed102-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.597589 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.597503 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2670c956-bd01-4574-b847-3a1b5c0ed102-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.597589 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.597521 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfdxx\" (UniqueName: \"kubernetes.io/projected/2670c956-bd01-4574-b847-3a1b5c0ed102-kube-api-access-pfdxx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.597699 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.597584 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2670c956-bd01-4574-b847-3a1b5c0ed102-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.597752 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.597688 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2670c956-bd01-4574-b847-3a1b5c0ed102-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.597752 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.597740 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2670c956-bd01-4574-b847-3a1b5c0ed102-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.647785 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.647759 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:33.647911 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.647809 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:33.662252 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.662228 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:33.698146 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.698117 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2670c956-bd01-4574-b847-3a1b5c0ed102-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.698262 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.698150 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2670c956-bd01-4574-b847-3a1b5c0ed102-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.698262 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.698174 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2670c956-bd01-4574-b847-3a1b5c0ed102-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.698262 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.698247 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2670c956-bd01-4574-b847-3a1b5c0ed102-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.698468 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.698325 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2670c956-bd01-4574-b847-3a1b5c0ed102-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.698468 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.698353 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfdxx\" (UniqueName: \"kubernetes.io/projected/2670c956-bd01-4574-b847-3a1b5c0ed102-kube-api-access-pfdxx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.698600 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.698576 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2670c956-bd01-4574-b847-3a1b5c0ed102-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.698695 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.698674 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2670c956-bd01-4574-b847-3a1b5c0ed102-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.698788 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.698767 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2670c956-bd01-4574-b847-3a1b5c0ed102-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.700396 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.700377 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2670c956-bd01-4574-b847-3a1b5c0ed102-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.700672 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.700654 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2670c956-bd01-4574-b847-3a1b5c0ed102-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.705498 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.705480 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfdxx\" (UniqueName: \"kubernetes.io/projected/2670c956-bd01-4574-b847-3a1b5c0ed102-kube-api-access-pfdxx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-crx9n\" (UID: \"2670c956-bd01-4574-b847-3a1b5c0ed102\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:33.883520 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:33.883491 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:34.154584 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:34.154552 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n"] Sep 08 18:49:34.155812 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:49:34.155784 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2670c956_bd01_4574_b847_3a1b5c0ed102.slice/crio-29315636d2dabd5dd0fc7863d208d07496c95c9245c843257fcc9a6fd9c6a1be WatchSource:0}: Error finding container 29315636d2dabd5dd0fc7863d208d07496c95c9245c843257fcc9a6fd9c6a1be: Status 404 returned error can't find the container with id 29315636d2dabd5dd0fc7863d208d07496c95c9245c843257fcc9a6fd9c6a1be Sep 08 18:49:34.456587 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:34.456497 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" event={"ID":"2670c956-bd01-4574-b847-3a1b5c0ed102","Type":"ContainerStarted","Data":"65e49bbe2e14eedfd322a7dce179be403155656af839c43142302528bfdac33d"} Sep 08 18:49:34.456587 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:34.456557 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" event={"ID":"2670c956-bd01-4574-b847-3a1b5c0ed102","Type":"ContainerStarted","Data":"29315636d2dabd5dd0fc7863d208d07496c95c9245c843257fcc9a6fd9c6a1be"} Sep 08 18:49:34.467942 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:34.467919 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-n7l5b" Sep 08 18:49:34.471608 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:34.471556 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" podStartSLOduration=1.319493323 podStartE2EDuration="1.471539078s" totalImagesPullingTime="152.045755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:49:34.471492715 +0000 UTC m=+910.177380946" watchObservedRunningTime="2026-09-08 18:49:34.471539078 +0000 UTC m=+910.177427311" Sep 08 18:49:43.884168 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:43.884121 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:43.885877 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:43.884204 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:43.896701 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:43.896680 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:44.519328 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:44.519293 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-crx9n" Sep 08 18:49:53.067242 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.067208 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf"] Sep 08 18:49:53.070089 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.070064 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.072923 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.072897 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 08 18:49:53.081702 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.081679 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf"] Sep 08 18:49:53.141584 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.141551 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b6cde27-cdeb-47e9-ad27-3e63042fd929-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.141723 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.141600 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6cde27-cdeb-47e9-ad27-3e63042fd929-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.141723 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.141680 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mp9h\" (UniqueName: \"kubernetes.io/projected/6b6cde27-cdeb-47e9-ad27-3e63042fd929-kube-api-access-6mp9h\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.141723 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.141707 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b6cde27-cdeb-47e9-ad27-3e63042fd929-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.141829 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.141728 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b6cde27-cdeb-47e9-ad27-3e63042fd929-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.141829 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.141746 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b6cde27-cdeb-47e9-ad27-3e63042fd929-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.242805 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.242762 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b6cde27-cdeb-47e9-ad27-3e63042fd929-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.242971 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.242820 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6cde27-cdeb-47e9-ad27-3e63042fd929-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.242971 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.242866 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mp9h\" (UniqueName: \"kubernetes.io/projected/6b6cde27-cdeb-47e9-ad27-3e63042fd929-kube-api-access-6mp9h\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.242971 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.242886 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b6cde27-cdeb-47e9-ad27-3e63042fd929-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.242971 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.242904 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b6cde27-cdeb-47e9-ad27-3e63042fd929-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.242971 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.242932 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b6cde27-cdeb-47e9-ad27-3e63042fd929-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.243278 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.243243 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b6cde27-cdeb-47e9-ad27-3e63042fd929-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.243278 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.243269 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b6cde27-cdeb-47e9-ad27-3e63042fd929-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.243488 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.243323 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b6cde27-cdeb-47e9-ad27-3e63042fd929-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.245196 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.245170 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b6cde27-cdeb-47e9-ad27-3e63042fd929-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.245356 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.245341 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b6cde27-cdeb-47e9-ad27-3e63042fd929-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.250506 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.250482 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mp9h\" (UniqueName: \"kubernetes.io/projected/6b6cde27-cdeb-47e9-ad27-3e63042fd929-kube-api-access-6mp9h\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2bmkf\" (UID: \"6b6cde27-cdeb-47e9-ad27-3e63042fd929\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.381634 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.381605 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:49:53.650878 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:53.650848 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf"] Sep 08 18:49:53.651932 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:49:53.651908 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6cde27_cdeb_47e9_ad27_3e63042fd929.slice/crio-d1e631dea72b419725096adae953f30212aa46be8f4ad101a67a723ed46a6f59 WatchSource:0}: Error finding container d1e631dea72b419725096adae953f30212aa46be8f4ad101a67a723ed46a6f59: Status 404 returned error can't find the container with id d1e631dea72b419725096adae953f30212aa46be8f4ad101a67a723ed46a6f59 Sep 08 18:49:54.557543 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:54.557502 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" event={"ID":"6b6cde27-cdeb-47e9-ad27-3e63042fd929","Type":"ContainerStarted","Data":"e309741868a7fe9b2e2d623207bbac9d4dec0af4482dc834e7ddb444b1fd5c6f"} Sep 08 18:49:54.557929 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:54.557572 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" event={"ID":"6b6cde27-cdeb-47e9-ad27-3e63042fd929","Type":"ContainerStarted","Data":"d1e631dea72b419725096adae953f30212aa46be8f4ad101a67a723ed46a6f59"} Sep 08 18:49:54.575474 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:49:54.575429 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" podStartSLOduration=1.3948023649999999 podStartE2EDuration="1.575414616s" totalImagesPullingTime="180.612251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:49:54.573511151 +0000 UTC m=+930.279399385" watchObservedRunningTime="2026-09-08 18:49:54.575414616 +0000 UTC m=+930.281302848" Sep 08 18:50:03.382713 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:50:03.382675 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:50:03.383162 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:50:03.382738 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:50:03.395330 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:50:03.395304 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:50:03.613760 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:50:03.613726 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2bmkf" Sep 08 18:51:54.153756 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.153718 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w"] Sep 08 18:51:54.157977 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.157958 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" Sep 08 18:51:54.164430 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.164099 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-52ba8b\"" Sep 08 18:51:54.164430 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.164221 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-52ba8b-dockercfg-ndz6t\"" Sep 08 18:51:54.178148 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.178118 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w"] Sep 08 18:51:54.241711 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.241675 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw"] Sep 08 18:51:54.245141 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.245122 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" Sep 08 18:51:54.260165 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.260140 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw"] Sep 08 18:51:54.268195 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.268170 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5rht\" (UniqueName: \"kubernetes.io/projected/4b2c3088-389c-4794-b7cd-aedcf837398e-kube-api-access-v5rht\") pod \"payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w\" (UID: \"4b2c3088-389c-4794-b7cd-aedcf837398e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" Sep 08 18:51:54.268342 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.268280 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2c3088-389c-4794-b7cd-aedcf837398e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w\" (UID: \"4b2c3088-389c-4794-b7cd-aedcf837398e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" Sep 08 18:51:54.268342 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.268336 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99bd63ad-bd5b-4e0b-ae93-70b20ab0310f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw\" (UID: \"99bd63ad-bd5b-4e0b-ae93-70b20ab0310f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" Sep 08 18:51:54.268471 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.268357 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlvnd\" (UniqueName: \"kubernetes.io/projected/99bd63ad-bd5b-4e0b-ae93-70b20ab0310f-kube-api-access-nlvnd\") pod \"payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw\" (UID: \"99bd63ad-bd5b-4e0b-ae93-70b20ab0310f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" Sep 08 18:51:54.369772 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.369734 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2c3088-389c-4794-b7cd-aedcf837398e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w\" (UID: \"4b2c3088-389c-4794-b7cd-aedcf837398e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" Sep 08 18:51:54.369942 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.369786 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99bd63ad-bd5b-4e0b-ae93-70b20ab0310f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw\" (UID: \"99bd63ad-bd5b-4e0b-ae93-70b20ab0310f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" Sep 08 18:51:54.369942 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.369805 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlvnd\" (UniqueName: \"kubernetes.io/projected/99bd63ad-bd5b-4e0b-ae93-70b20ab0310f-kube-api-access-nlvnd\") pod \"payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw\" (UID: \"99bd63ad-bd5b-4e0b-ae93-70b20ab0310f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" Sep 08 18:51:54.369942 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.369833 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5rht\" (UniqueName: \"kubernetes.io/projected/4b2c3088-389c-4794-b7cd-aedcf837398e-kube-api-access-v5rht\") pod \"payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w\" (UID: \"4b2c3088-389c-4794-b7cd-aedcf837398e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" Sep 08 18:51:54.370599 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.370573 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99bd63ad-bd5b-4e0b-ae93-70b20ab0310f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw\" (UID: \"99bd63ad-bd5b-4e0b-ae93-70b20ab0310f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" Sep 08 18:51:54.370695 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.370619 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2c3088-389c-4794-b7cd-aedcf837398e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w\" (UID: \"4b2c3088-389c-4794-b7cd-aedcf837398e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" Sep 08 18:51:54.381975 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.381942 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5rht\" (UniqueName: \"kubernetes.io/projected/4b2c3088-389c-4794-b7cd-aedcf837398e-kube-api-access-v5rht\") pod \"payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w\" (UID: \"4b2c3088-389c-4794-b7cd-aedcf837398e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" Sep 08 18:51:54.385840 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.385819 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlvnd\" (UniqueName: \"kubernetes.io/projected/99bd63ad-bd5b-4e0b-ae93-70b20ab0310f-kube-api-access-nlvnd\") pod \"payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw\" (UID: \"99bd63ad-bd5b-4e0b-ae93-70b20ab0310f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" Sep 08 18:51:54.468715 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.468640 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" Sep 08 18:51:54.554776 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.554721 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" Sep 08 18:51:54.758434 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.758410 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w"] Sep 08 18:51:54.759850 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:51:54.759826 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2c3088_389c_4794_b7cd_aedcf837398e.slice/crio-d7d6692603d9d97711c23e3abb461f4bf2076c077a1656fc4fe7861e14b7042a WatchSource:0}: Error finding container d7d6692603d9d97711c23e3abb461f4bf2076c077a1656fc4fe7861e14b7042a: Status 404 returned error can't find the container with id d7d6692603d9d97711c23e3abb461f4bf2076c077a1656fc4fe7861e14b7042a Sep 08 18:51:54.847693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:54.847669 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw"] Sep 08 18:51:54.849260 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:51:54.849219 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bd63ad_bd5b_4e0b_ae93_70b20ab0310f.slice/crio-302a6aaf989fb3aa3adcb150a4b78c87ba61449dce3a6b9c50aeff13b8880a44 WatchSource:0}: Error finding container 302a6aaf989fb3aa3adcb150a4b78c87ba61449dce3a6b9c50aeff13b8880a44: Status 404 returned error can't find the container with id 302a6aaf989fb3aa3adcb150a4b78c87ba61449dce3a6b9c50aeff13b8880a44 Sep 08 18:51:55.166001 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:55.165964 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" event={"ID":"4b2c3088-389c-4794-b7cd-aedcf837398e","Type":"ContainerStarted","Data":"629d108319c2aebac5eb1451e50b12ac1d756a43f4fc53ea13a07a4cd39aeaf8"} Sep 08 18:51:55.166487 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:55.166026 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" event={"ID":"4b2c3088-389c-4794-b7cd-aedcf837398e","Type":"ContainerStarted","Data":"d7d6692603d9d97711c23e3abb461f4bf2076c077a1656fc4fe7861e14b7042a"} Sep 08 18:51:55.166487 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:55.166056 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" Sep 08 18:51:55.167832 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:55.167802 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" event={"ID":"99bd63ad-bd5b-4e0b-ae93-70b20ab0310f","Type":"ContainerStarted","Data":"5603c09c604cc0b098ae045748ee994de193a22680f8e20d63dd97956a5e37bb"} Sep 08 18:51:55.167960 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:55.167853 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" event={"ID":"99bd63ad-bd5b-4e0b-ae93-70b20ab0310f","Type":"ContainerStarted","Data":"302a6aaf989fb3aa3adcb150a4b78c87ba61449dce3a6b9c50aeff13b8880a44"} Sep 08 18:51:55.167960 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:55.167884 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" Sep 08 18:51:55.224959 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:55.224906 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" podStartSLOduration=1.224886847 podStartE2EDuration="1.224886847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:51:55.224306818 +0000 UTC m=+1050.930195051" watchObservedRunningTime="2026-09-08 18:51:55.224886847 +0000 UTC m=+1050.930775080" Sep 08 18:51:55.261215 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:55.261167 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" podStartSLOduration=1.2611537130000001 podStartE2EDuration="1.261153713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:51:55.259305048 +0000 UTC m=+1050.965193282" watchObservedRunningTime="2026-09-08 18:51:55.261153713 +0000 UTC m=+1050.967041946" Sep 08 18:51:58.099678 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.099645 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx"] Sep 08 18:51:58.103280 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.103255 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.107335 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.107312 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ffda9270-openshift-default-dockercfg-24gnk\"" Sep 08 18:51:58.132666 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.132640 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx"] Sep 08 18:51:58.204012 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.203982 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-workload-socket\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.204180 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.204027 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-podinfo\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.204180 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.204048 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-envoy\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.204180 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.204064 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-token\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.204180 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.204158 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-workload-certs\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.204456 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.204224 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istiod-ca-cert\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.204456 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.204256 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-credential-socket\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.204456 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.204279 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-data\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.204456 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.204327 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg9w7\" (UniqueName: \"kubernetes.io/projected/66b6a2fc-2a2d-41a2-9455-d9a305177d33-kube-api-access-jg9w7\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.305461 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.305421 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-podinfo\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.305646 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.305466 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-envoy\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.305646 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.305492 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-token\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.305646 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.305552 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-workload-certs\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.305646 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.305602 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istiod-ca-cert\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.305646 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.305625 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-credential-socket\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.305905 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.305647 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-data\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.305905 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.305679 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg9w7\" (UniqueName: \"kubernetes.io/projected/66b6a2fc-2a2d-41a2-9455-d9a305177d33-kube-api-access-jg9w7\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.305905 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.305745 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-workload-socket\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.306080 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.306054 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-workload-certs\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.306139 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.306096 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-credential-socket\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.306224 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.306206 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-data\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.306339 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.306318 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istiod-ca-cert\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.306418 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.306324 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-workload-socket\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.307920 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.307895 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-envoy\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.308074 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.308059 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-podinfo\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.316599 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.316575 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-token\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.317317 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.317301 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg9w7\" (UniqueName: \"kubernetes.io/projected/66b6a2fc-2a2d-41a2-9455-d9a305177d33-kube-api-access-jg9w7\") pod \"e2e-ait-ffda9270-openshift-default-744696f866-6g7qx\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.414529 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.414505 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:58.701632 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.701605 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx"] Sep 08 18:51:58.702838 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:51:58.702813 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b6a2fc_2a2d_41a2_9455_d9a305177d33.slice/crio-d2947f08d77253c5d54d1716b7acac42446f813eed19f19ac7faf63e36c652bc WatchSource:0}: Error finding container d2947f08d77253c5d54d1716b7acac42446f813eed19f19ac7faf63e36c652bc: Status 404 returned error can't find the container with id d2947f08d77253c5d54d1716b7acac42446f813eed19f19ac7faf63e36c652bc Sep 08 18:51:58.705062 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.705023 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:51:58.705160 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.705104 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:51:58.705160 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:58.705150 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:51:59.194724 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:59.194683 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" event={"ID":"66b6a2fc-2a2d-41a2-9455-d9a305177d33","Type":"ContainerStarted","Data":"7cf36c47f9ee15979e5511bce18f3eb36cacce60c37f9f4f7e04a4275b311171"} Sep 08 18:51:59.195129 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:59.194740 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" event={"ID":"66b6a2fc-2a2d-41a2-9455-d9a305177d33","Type":"ContainerStarted","Data":"d2947f08d77253c5d54d1716b7acac42446f813eed19f19ac7faf63e36c652bc"} Sep 08 18:51:59.219297 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:59.219250 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" podStartSLOduration=1.219235515 podStartE2EDuration="1.219235515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:51:59.216723804 +0000 UTC m=+1054.922612049" watchObservedRunningTime="2026-09-08 18:51:59.219235515 +0000 UTC m=+1054.925123747" Sep 08 18:51:59.415352 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:59.415315 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:51:59.420306 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:51:59.420277 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:52:00.201139 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:00.201107 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:52:00.201970 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:00.201950 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:52:02.544171 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:02.544135 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt"] Sep 08 18:52:02.548885 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:02.548864 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" Sep 08 18:52:02.551289 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:02.551266 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ffda9270-dockercfg-vjwfd\"" Sep 08 18:52:02.551560 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:02.551538 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ffda9270\"" Sep 08 18:52:02.565281 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:02.565255 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt"] Sep 08 18:52:02.646434 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:02.646403 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433527c6-cda0-4dbd-bb6c-0f2455dbb588-plugins-config-volume\") pod \"payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt\" (UID: \"433527c6-cda0-4dbd-bb6c-0f2455dbb588\") " pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" Sep 08 18:52:02.646587 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:02.646481 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wgnd\" (UniqueName: \"kubernetes.io/projected/433527c6-cda0-4dbd-bb6c-0f2455dbb588-kube-api-access-8wgnd\") pod \"payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt\" (UID: \"433527c6-cda0-4dbd-bb6c-0f2455dbb588\") " pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" Sep 08 18:52:02.747064 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:02.747019 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wgnd\" (UniqueName: \"kubernetes.io/projected/433527c6-cda0-4dbd-bb6c-0f2455dbb588-kube-api-access-8wgnd\") pod \"payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt\" (UID: \"433527c6-cda0-4dbd-bb6c-0f2455dbb588\") " pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" Sep 08 18:52:02.747260 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:02.747140 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433527c6-cda0-4dbd-bb6c-0f2455dbb588-plugins-config-volume\") pod \"payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt\" (UID: \"433527c6-cda0-4dbd-bb6c-0f2455dbb588\") " pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" Sep 08 18:52:02.748012 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:02.747985 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433527c6-cda0-4dbd-bb6c-0f2455dbb588-plugins-config-volume\") pod \"payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt\" (UID: \"433527c6-cda0-4dbd-bb6c-0f2455dbb588\") " pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" Sep 08 18:52:02.761848 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:02.761817 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wgnd\" (UniqueName: \"kubernetes.io/projected/433527c6-cda0-4dbd-bb6c-0f2455dbb588-kube-api-access-8wgnd\") pod \"payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt\" (UID: \"433527c6-cda0-4dbd-bb6c-0f2455dbb588\") " pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" Sep 08 18:52:02.860646 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:02.860559 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" Sep 08 18:52:03.134543 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:03.134489 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt"] Sep 08 18:52:03.135321 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:52:03.135295 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433527c6_cda0_4dbd_bb6c_0f2455dbb588.slice/crio-3b54e26952b229783d80ed1464a17433d7fdfb801eeb4ada8d6da55cab09ce6d WatchSource:0}: Error finding container 3b54e26952b229783d80ed1464a17433d7fdfb801eeb4ada8d6da55cab09ce6d: Status 404 returned error can't find the container with id 3b54e26952b229783d80ed1464a17433d7fdfb801eeb4ada8d6da55cab09ce6d Sep 08 18:52:03.217805 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:03.217776 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" event={"ID":"433527c6-cda0-4dbd-bb6c-0f2455dbb588","Type":"ContainerStarted","Data":"146da99c8c57524357ee9324774c90f1611b16cb5f9f68b2fc8b4b81d9edfe3e"} Sep 08 18:52:03.217935 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:03.217832 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" event={"ID":"433527c6-cda0-4dbd-bb6c-0f2455dbb588","Type":"ContainerStarted","Data":"3b54e26952b229783d80ed1464a17433d7fdfb801eeb4ada8d6da55cab09ce6d"} Sep 08 18:52:03.217935 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:03.217887 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" Sep 08 18:52:03.239831 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:03.239777 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" podStartSLOduration=1.239761323 podStartE2EDuration="1.239761323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:03.237610243 +0000 UTC m=+1058.943498472" watchObservedRunningTime="2026-09-08 18:52:03.239761323 +0000 UTC m=+1058.945649578" Sep 08 18:52:05.869689 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:05.869655 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh"] Sep 08 18:52:05.873327 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:05.873308 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" Sep 08 18:52:05.876102 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:05.876082 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-235f65\"" Sep 08 18:52:05.876356 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:05.876340 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-235f65-dockercfg-rph7h\"" Sep 08 18:52:05.891186 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:05.891162 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh"] Sep 08 18:52:05.974202 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:05.974169 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876fc381-1574-468c-803b-5e6aa3bc311d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh\" (UID: \"876fc381-1574-468c-803b-5e6aa3bc311d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" Sep 08 18:52:05.974202 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:05.974204 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74ksj\" (UniqueName: \"kubernetes.io/projected/876fc381-1574-468c-803b-5e6aa3bc311d-kube-api-access-74ksj\") pod \"payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh\" (UID: \"876fc381-1574-468c-803b-5e6aa3bc311d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" Sep 08 18:52:06.075041 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:06.075007 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876fc381-1574-468c-803b-5e6aa3bc311d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh\" (UID: \"876fc381-1574-468c-803b-5e6aa3bc311d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" Sep 08 18:52:06.075041 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:06.075043 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74ksj\" (UniqueName: \"kubernetes.io/projected/876fc381-1574-468c-803b-5e6aa3bc311d-kube-api-access-74ksj\") pod \"payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh\" (UID: \"876fc381-1574-468c-803b-5e6aa3bc311d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" Sep 08 18:52:06.075768 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:06.075745 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876fc381-1574-468c-803b-5e6aa3bc311d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh\" (UID: \"876fc381-1574-468c-803b-5e6aa3bc311d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" Sep 08 18:52:06.085710 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:06.085680 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ksj\" (UniqueName: \"kubernetes.io/projected/876fc381-1574-468c-803b-5e6aa3bc311d-kube-api-access-74ksj\") pod \"payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh\" (UID: \"876fc381-1574-468c-803b-5e6aa3bc311d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" Sep 08 18:52:06.177399 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:06.177351 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" Sep 08 18:52:06.177549 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:06.177467 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" Sep 08 18:52:06.183827 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:06.183804 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" Sep 08 18:52:06.474120 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:06.474091 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh"] Sep 08 18:52:06.475335 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:52:06.475297 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod876fc381_1574_468c_803b_5e6aa3bc311d.slice/crio-76fc803d2b6e6ecd0bccc88f204d663c8cba39969937de649497a8bb2c4f7739 WatchSource:0}: Error finding container 76fc803d2b6e6ecd0bccc88f204d663c8cba39969937de649497a8bb2c4f7739: Status 404 returned error can't find the container with id 76fc803d2b6e6ecd0bccc88f204d663c8cba39969937de649497a8bb2c4f7739 Sep 08 18:52:07.246823 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:07.246783 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" event={"ID":"876fc381-1574-468c-803b-5e6aa3bc311d","Type":"ContainerStarted","Data":"92a02ddc61d0f8fbbce63f9556837770a24e024dbbc0a522004a793a8008f71f"} Sep 08 18:52:07.247293 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:07.246855 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" Sep 08 18:52:07.247293 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:07.246884 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" event={"ID":"876fc381-1574-468c-803b-5e6aa3bc311d","Type":"ContainerStarted","Data":"76fc803d2b6e6ecd0bccc88f204d663c8cba39969937de649497a8bb2c4f7739"} Sep 08 18:52:07.264343 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:07.264287 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" podStartSLOduration=2.264266062 podStartE2EDuration="2.264266062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:07.262894941 +0000 UTC m=+1062.968783172" watchObservedRunningTime="2026-09-08 18:52:07.264266062 +0000 UTC m=+1062.970154296" Sep 08 18:52:14.227604 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:14.227568 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" Sep 08 18:52:16.152287 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:16.152256 2553 patch_prober.go:28] interesting pod/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": read tcp 10.134.0.2:57648->10.134.0.47:15021: read: connection reset by peer" start-of-body= Sep 08 18:52:16.152711 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:16.152319 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" podUID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": read tcp 10.134.0.2:57648->10.134.0.47:15021: read: connection reset by peer" Sep 08 18:52:16.310767 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:16.310687 2553 generic.go:356] "Generic (PLEG): container finished" podID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerID="7cf36c47f9ee15979e5511bce18f3eb36cacce60c37f9f4f7e04a4275b311171" exitCode=0 Sep 08 18:52:16.310928 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:16.310761 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" event={"ID":"66b6a2fc-2a2d-41a2-9455-d9a305177d33","Type":"ContainerDied","Data":"7cf36c47f9ee15979e5511bce18f3eb36cacce60c37f9f4f7e04a4275b311171"} Sep 08 18:52:16.311235 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:16.311216 2553 scope.go:117] "RemoveContainer" containerID="7cf36c47f9ee15979e5511bce18f3eb36cacce60c37f9f4f7e04a4275b311171" Sep 08 18:52:16.312259 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:16.312230 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:52:16.312314 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:16.312296 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:52:16.312377 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:16.312330 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:52:17.320490 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:17.320400 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" event={"ID":"66b6a2fc-2a2d-41a2-9455-d9a305177d33","Type":"ContainerStarted","Data":"e38616a674afc7a8856cfea2c7004e66979a9235f3aa08e04d2204e4059e8c4f"} Sep 08 18:52:17.415001 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:17.414964 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:52:17.415184 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:17.415136 2553 patch_prober.go:28] interesting pod/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 08 18:52:17.415239 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:17.415210 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" podUID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 08 18:52:18.266731 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:18.266695 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" Sep 08 18:52:18.414934 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:18.414903 2553 patch_prober.go:28] interesting pod/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 08 18:52:18.415286 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:18.414965 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" podUID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 08 18:52:19.415270 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:19.415228 2553 patch_prober.go:28] interesting pod/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 08 18:52:19.415672 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:19.415307 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" podUID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 08 18:52:20.415241 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:20.415204 2553 patch_prober.go:28] interesting pod/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 08 18:52:20.415470 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:20.415280 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" podUID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 08 18:52:21.415441 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:21.415408 2553 patch_prober.go:28] interesting pod/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 08 18:52:21.415619 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:21.415462 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" podUID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 08 18:52:22.415181 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.415141 2553 patch_prober.go:28] interesting pod/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 08 18:52:22.415424 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.415387 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" podUID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 08 18:52:22.705964 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.705882 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44"] Sep 08 18:52:22.708275 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.708257 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" Sep 08 18:52:22.710967 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.710947 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9c3074-dockercfg-gg748\"" Sep 08 18:52:22.713851 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.713830 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9c3074\"" Sep 08 18:52:22.722010 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.721988 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44"] Sep 08 18:52:22.732922 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.732898 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j"] Sep 08 18:52:22.735332 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.735314 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" Sep 08 18:52:22.749177 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.749151 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j"] Sep 08 18:52:22.829182 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.829145 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834b0555-3f80-49de-a999-93e66bad8521-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44\" (UID: \"834b0555-3f80-49de-a999-93e66bad8521\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" Sep 08 18:52:22.829352 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.829206 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643a9a33-2c05-47a9-9bd3-fb58abfa8705-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j\" (UID: \"643a9a33-2c05-47a9-9bd3-fb58abfa8705\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" Sep 08 18:52:22.829352 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.829231 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htf86\" (UniqueName: \"kubernetes.io/projected/643a9a33-2c05-47a9-9bd3-fb58abfa8705-kube-api-access-htf86\") pod \"payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j\" (UID: \"643a9a33-2c05-47a9-9bd3-fb58abfa8705\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" Sep 08 18:52:22.829352 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.829316 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdl86\" (UniqueName: \"kubernetes.io/projected/834b0555-3f80-49de-a999-93e66bad8521-kube-api-access-jdl86\") pod \"payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44\" (UID: \"834b0555-3f80-49de-a999-93e66bad8521\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" Sep 08 18:52:22.930814 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.930783 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdl86\" (UniqueName: \"kubernetes.io/projected/834b0555-3f80-49de-a999-93e66bad8521-kube-api-access-jdl86\") pod \"payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44\" (UID: \"834b0555-3f80-49de-a999-93e66bad8521\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" Sep 08 18:52:22.931011 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.930858 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834b0555-3f80-49de-a999-93e66bad8521-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44\" (UID: \"834b0555-3f80-49de-a999-93e66bad8521\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" Sep 08 18:52:22.931011 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.930900 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643a9a33-2c05-47a9-9bd3-fb58abfa8705-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j\" (UID: \"643a9a33-2c05-47a9-9bd3-fb58abfa8705\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" Sep 08 18:52:22.931011 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.930923 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htf86\" (UniqueName: \"kubernetes.io/projected/643a9a33-2c05-47a9-9bd3-fb58abfa8705-kube-api-access-htf86\") pod \"payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j\" (UID: \"643a9a33-2c05-47a9-9bd3-fb58abfa8705\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" Sep 08 18:52:22.931872 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.931842 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643a9a33-2c05-47a9-9bd3-fb58abfa8705-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j\" (UID: \"643a9a33-2c05-47a9-9bd3-fb58abfa8705\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" Sep 08 18:52:22.931996 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.931963 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834b0555-3f80-49de-a999-93e66bad8521-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44\" (UID: \"834b0555-3f80-49de-a999-93e66bad8521\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" Sep 08 18:52:22.939175 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.939149 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htf86\" (UniqueName: \"kubernetes.io/projected/643a9a33-2c05-47a9-9bd3-fb58abfa8705-kube-api-access-htf86\") pod \"payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j\" (UID: \"643a9a33-2c05-47a9-9bd3-fb58abfa8705\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" Sep 08 18:52:22.939589 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:22.939564 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdl86\" (UniqueName: \"kubernetes.io/projected/834b0555-3f80-49de-a999-93e66bad8521-kube-api-access-jdl86\") pod \"payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44\" (UID: \"834b0555-3f80-49de-a999-93e66bad8521\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" Sep 08 18:52:23.018640 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:23.018553 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" Sep 08 18:52:23.047901 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:23.047871 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" Sep 08 18:52:23.291309 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:23.291273 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44"] Sep 08 18:52:23.292395 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:52:23.292350 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834b0555_3f80_49de_a999_93e66bad8521.slice/crio-5818aff4805315219f39e616ef41a077d6458e00201b4e72d6cd60d5e928ea34 WatchSource:0}: Error finding container 5818aff4805315219f39e616ef41a077d6458e00201b4e72d6cd60d5e928ea34: Status 404 returned error can't find the container with id 5818aff4805315219f39e616ef41a077d6458e00201b4e72d6cd60d5e928ea34 Sep 08 18:52:23.327264 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:23.327241 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j"] Sep 08 18:52:23.328689 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:52:23.328665 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643a9a33_2c05_47a9_9bd3_fb58abfa8705.slice/crio-8834a462eab7b1b794d0ad00bec3b6c8692629a6ecc01ddbf8211c529749ebc6 WatchSource:0}: Error finding container 8834a462eab7b1b794d0ad00bec3b6c8692629a6ecc01ddbf8211c529749ebc6: Status 404 returned error can't find the container with id 8834a462eab7b1b794d0ad00bec3b6c8692629a6ecc01ddbf8211c529749ebc6 Sep 08 18:52:23.364558 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:23.364523 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" event={"ID":"643a9a33-2c05-47a9-9bd3-fb58abfa8705","Type":"ContainerStarted","Data":"8834a462eab7b1b794d0ad00bec3b6c8692629a6ecc01ddbf8211c529749ebc6"} Sep 08 18:52:23.366295 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:23.366271 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" event={"ID":"834b0555-3f80-49de-a999-93e66bad8521","Type":"ContainerStarted","Data":"e0503a20e8735d484aff4889e2b39f3e066aec22a6aa904d8f6d08f65541d1e7"} Sep 08 18:52:23.366403 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:23.366319 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" event={"ID":"834b0555-3f80-49de-a999-93e66bad8521","Type":"ContainerStarted","Data":"5818aff4805315219f39e616ef41a077d6458e00201b4e72d6cd60d5e928ea34"} Sep 08 18:52:23.366464 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:23.366403 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" Sep 08 18:52:23.385165 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:23.385120 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" podStartSLOduration=1.385103295 podStartE2EDuration="1.385103295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:23.382786907 +0000 UTC m=+1079.088675141" watchObservedRunningTime="2026-09-08 18:52:23.385103295 +0000 UTC m=+1079.090991527" Sep 08 18:52:23.419540 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:23.419507 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:52:23.419925 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:23.419904 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:52:23.420694 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:23.420674 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:52:24.373822 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:24.373789 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" event={"ID":"643a9a33-2c05-47a9-9bd3-fb58abfa8705","Type":"ContainerStarted","Data":"cff8c08e9ab1fb3a8e7fa7cf6aee41fe1d8e2f43508e0efc5d16f13e2a7c621c"} Sep 08 18:52:24.397243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:24.397183 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" podStartSLOduration=2.397166939 podStartE2EDuration="2.397166939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:24.394936747 +0000 UTC m=+1080.100824981" watchObservedRunningTime="2026-09-08 18:52:24.397166939 +0000 UTC m=+1080.103055170" Sep 08 18:52:25.380176 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:25.380139 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" Sep 08 18:52:26.272005 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:26.271969 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffda9270-fp7md"] Sep 08 18:52:26.275324 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:26.275302 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffda9270-fp7md" Sep 08 18:52:26.277986 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:26.277963 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 18:52:26.278100 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:26.277984 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 18:52:26.284138 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:26.284118 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffda9270-fp7md"] Sep 08 18:52:26.361118 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:26.361080 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebeb885-8556-444d-9383-61bbf34a55ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffda9270-fp7md\" (UID: \"cebeb885-8556-444d-9383-61bbf34a55ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffda9270-fp7md" Sep 08 18:52:26.461791 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:26.461752 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebeb885-8556-444d-9383-61bbf34a55ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffda9270-fp7md\" (UID: \"cebeb885-8556-444d-9383-61bbf34a55ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffda9270-fp7md" Sep 08 18:52:26.462467 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:26.462443 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebeb885-8556-444d-9383-61bbf34a55ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffda9270-fp7md\" (UID: \"cebeb885-8556-444d-9383-61bbf34a55ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffda9270-fp7md" Sep 08 18:52:26.588005 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:26.587917 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffda9270-fp7md" Sep 08 18:52:26.875866 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:26.875834 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffda9270-fp7md"] Sep 08 18:52:26.876758 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:52:26.876736 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebeb885_8556_444d_9383_61bbf34a55ba.slice/crio-55aecba527534ca4af9fd8b27420efff6566b85f3f34055496846679fb996937 WatchSource:0}: Error finding container 55aecba527534ca4af9fd8b27420efff6566b85f3f34055496846679fb996937: Status 404 returned error can't find the container with id 55aecba527534ca4af9fd8b27420efff6566b85f3f34055496846679fb996937 Sep 08 18:52:26.878933 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:26.878907 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 18:52:27.393293 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:27.393256 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffda9270-fp7md" event={"ID":"cebeb885-8556-444d-9383-61bbf34a55ba","Type":"ContainerStarted","Data":"55aecba527534ca4af9fd8b27420efff6566b85f3f34055496846679fb996937"} Sep 08 18:52:30.415574 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:30.415544 2553 generic.go:356] "Generic (PLEG): container finished" podID="cebeb885-8556-444d-9383-61bbf34a55ba" containerID="4d6b0e63b2cbd671f1b7290911196f33694d442f1584db6cd43b7ef83fcc40ad" exitCode=0 Sep 08 18:52:30.415925 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:30.415634 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffda9270-fp7md" event={"ID":"cebeb885-8556-444d-9383-61bbf34a55ba","Type":"ContainerDied","Data":"4d6b0e63b2cbd671f1b7290911196f33694d442f1584db6cd43b7ef83fcc40ad"} Sep 08 18:52:31.542691 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:31.542669 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffda9270-fp7md" Sep 08 18:52:31.715690 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:31.715606 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebeb885-8556-444d-9383-61bbf34a55ba-maas-api-service-ca\") pod \"cebeb885-8556-444d-9383-61bbf34a55ba\" (UID: \"cebeb885-8556-444d-9383-61bbf34a55ba\") " Sep 08 18:52:31.715945 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:31.715911 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cebeb885-8556-444d-9383-61bbf34a55ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cebeb885-8556-444d-9383-61bbf34a55ba" (UID: "cebeb885-8556-444d-9383-61bbf34a55ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:52:31.817032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:31.816992 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebeb885-8556-444d-9383-61bbf34a55ba-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:52:32.427987 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:32.427960 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffda9270-fp7md" Sep 08 18:52:32.428160 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:32.427997 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffda9270-fp7md" event={"ID":"cebeb885-8556-444d-9383-61bbf34a55ba","Type":"ContainerDied","Data":"55aecba527534ca4af9fd8b27420efff6566b85f3f34055496846679fb996937"} Sep 08 18:52:32.428160 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:32.428042 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55aecba527534ca4af9fd8b27420efff6566b85f3f34055496846679fb996937" Sep 08 18:52:34.376269 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:34.376235 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" Sep 08 18:52:36.388550 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:36.388517 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" Sep 08 18:52:42.631823 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:42.631784 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt"] Sep 08 18:52:42.632408 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:42.632087 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" podUID="433527c6-cda0-4dbd-bb6c-0f2455dbb588" containerName="payload-processing" containerID="cri-o://146da99c8c57524357ee9324774c90f1611b16cb5f9f68b2fc8b4b81d9edfe3e" gracePeriod=30 Sep 08 18:52:42.835666 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:42.835609 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffda9270-fp7md"] Sep 08 18:52:42.838034 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:42.838005 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffda9270-fp7md"] Sep 08 18:52:42.892152 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:42.892089 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" Sep 08 18:52:42.909175 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:42.909147 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433527c6-cda0-4dbd-bb6c-0f2455dbb588-plugins-config-volume\") pod \"433527c6-cda0-4dbd-bb6c-0f2455dbb588\" (UID: \"433527c6-cda0-4dbd-bb6c-0f2455dbb588\") " Sep 08 18:52:42.909356 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:42.909204 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wgnd\" (UniqueName: \"kubernetes.io/projected/433527c6-cda0-4dbd-bb6c-0f2455dbb588-kube-api-access-8wgnd\") pod \"433527c6-cda0-4dbd-bb6c-0f2455dbb588\" (UID: \"433527c6-cda0-4dbd-bb6c-0f2455dbb588\") " Sep 08 18:52:42.909727 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:42.909697 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433527c6-cda0-4dbd-bb6c-0f2455dbb588-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "433527c6-cda0-4dbd-bb6c-0f2455dbb588" (UID: "433527c6-cda0-4dbd-bb6c-0f2455dbb588"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:52:42.912194 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:42.912167 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433527c6-cda0-4dbd-bb6c-0f2455dbb588-kube-api-access-8wgnd" (OuterVolumeSpecName: "kube-api-access-8wgnd") pod "433527c6-cda0-4dbd-bb6c-0f2455dbb588" (UID: "433527c6-cda0-4dbd-bb6c-0f2455dbb588"). InnerVolumeSpecName "kube-api-access-8wgnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:52:42.976376 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:42.976326 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx"] Sep 08 18:52:42.976876 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:42.976812 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" podUID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerName="istio-proxy" containerID="cri-o://e38616a674afc7a8856cfea2c7004e66979a9235f3aa08e04d2204e4059e8c4f" gracePeriod=30 Sep 08 18:52:43.010744 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:43.010716 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433527c6-cda0-4dbd-bb6c-0f2455dbb588-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:52:43.010744 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:43.010740 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wgnd\" (UniqueName: \"kubernetes.io/projected/433527c6-cda0-4dbd-bb6c-0f2455dbb588-kube-api-access-8wgnd\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:52:43.494929 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:43.494890 2553 generic.go:356] "Generic (PLEG): container finished" podID="433527c6-cda0-4dbd-bb6c-0f2455dbb588" containerID="146da99c8c57524357ee9324774c90f1611b16cb5f9f68b2fc8b4b81d9edfe3e" exitCode=0 Sep 08 18:52:43.494929 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:43.494927 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" Sep 08 18:52:43.495155 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:43.494974 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" event={"ID":"433527c6-cda0-4dbd-bb6c-0f2455dbb588","Type":"ContainerDied","Data":"146da99c8c57524357ee9324774c90f1611b16cb5f9f68b2fc8b4b81d9edfe3e"} Sep 08 18:52:43.495155 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:43.495022 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt" event={"ID":"433527c6-cda0-4dbd-bb6c-0f2455dbb588","Type":"ContainerDied","Data":"3b54e26952b229783d80ed1464a17433d7fdfb801eeb4ada8d6da55cab09ce6d"} Sep 08 18:52:43.495155 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:43.495043 2553 scope.go:117] "RemoveContainer" containerID="146da99c8c57524357ee9324774c90f1611b16cb5f9f68b2fc8b4b81d9edfe3e" Sep 08 18:52:43.505892 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:43.505873 2553 scope.go:117] "RemoveContainer" containerID="146da99c8c57524357ee9324774c90f1611b16cb5f9f68b2fc8b4b81d9edfe3e" Sep 08 18:52:43.506181 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:52:43.506161 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"146da99c8c57524357ee9324774c90f1611b16cb5f9f68b2fc8b4b81d9edfe3e\": container with ID starting with 146da99c8c57524357ee9324774c90f1611b16cb5f9f68b2fc8b4b81d9edfe3e not found: ID does not exist" containerID="146da99c8c57524357ee9324774c90f1611b16cb5f9f68b2fc8b4b81d9edfe3e" Sep 08 18:52:43.506250 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:43.506194 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"146da99c8c57524357ee9324774c90f1611b16cb5f9f68b2fc8b4b81d9edfe3e"} err="failed to get container status \"146da99c8c57524357ee9324774c90f1611b16cb5f9f68b2fc8b4b81d9edfe3e\": rpc error: code = NotFound desc = could not find container \"146da99c8c57524357ee9324774c90f1611b16cb5f9f68b2fc8b4b81d9edfe3e\": container with ID starting with 146da99c8c57524357ee9324774c90f1611b16cb5f9f68b2fc8b4b81d9edfe3e not found: ID does not exist" Sep 08 18:52:43.521596 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:43.521567 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt"] Sep 08 18:52:43.529479 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:43.529451 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ffda9270-68cd44d7df-bkbbt"] Sep 08 18:52:44.004784 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.004750 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh"] Sep 08 18:52:44.005405 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.005382 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433527c6-cda0-4dbd-bb6c-0f2455dbb588" containerName="payload-processing" Sep 08 18:52:44.005461 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.005408 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="433527c6-cda0-4dbd-bb6c-0f2455dbb588" containerName="payload-processing" Sep 08 18:52:44.005461 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.005442 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cebeb885-8556-444d-9383-61bbf34a55ba" containerName="revoke-keys" Sep 08 18:52:44.005461 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.005452 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cebeb885-8556-444d-9383-61bbf34a55ba" containerName="revoke-keys" Sep 08 18:52:44.005587 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.005574 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="433527c6-cda0-4dbd-bb6c-0f2455dbb588" containerName="payload-processing" Sep 08 18:52:44.005624 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.005596 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cebeb885-8556-444d-9383-61bbf34a55ba" containerName="revoke-keys" Sep 08 18:52:44.013069 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.013047 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.016227 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.016197 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-62eb1773-gw-openshift-default-dockercfg-jqw5n\"" Sep 08 18:52:44.017482 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.017461 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-workload-certs\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.017574 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.017495 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-token\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.017574 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.017515 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-workload-socket\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.017669 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.017630 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbwhc\" (UniqueName: \"kubernetes.io/projected/a25567e8-7d2d-4f0e-9551-a4054118dfd6-kube-api-access-rbwhc\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.017719 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.017684 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istiod-ca-cert\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.017719 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.017711 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-envoy\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.017806 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.017778 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-data\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.017840 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.017803 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-credential-socket\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.017886 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.017850 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-podinfo\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.023482 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.023439 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh"] Sep 08 18:52:44.119991 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.119938 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-workload-certs\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.120256 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.120236 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-token\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.120419 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.120404 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-workload-socket\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.121091 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.120588 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-workload-certs\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.121091 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.120592 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbwhc\" (UniqueName: \"kubernetes.io/projected/a25567e8-7d2d-4f0e-9551-a4054118dfd6-kube-api-access-rbwhc\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.121091 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.120671 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istiod-ca-cert\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.121091 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.120708 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-envoy\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.121091 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.120750 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-data\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.121091 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.120777 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-credential-socket\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.121091 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.120799 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-workload-socket\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.121091 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.120832 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-podinfo\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.121091 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.121051 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-data\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.121582 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.121106 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-credential-socket\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.121582 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.121322 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istiod-ca-cert\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.123326 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.123303 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-envoy\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.123979 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.123956 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-podinfo\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.138286 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.138261 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbwhc\" (UniqueName: \"kubernetes.io/projected/a25567e8-7d2d-4f0e-9551-a4054118dfd6-kube-api-access-rbwhc\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.139541 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.139520 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-token\") pod \"e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.325845 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.325767 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:44.605854 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.605821 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh"] Sep 08 18:52:44.606857 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:52:44.606832 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25567e8_7d2d_4f0e_9551_a4054118dfd6.slice/crio-cae2fa77888b0f263d6766157c52f9c4dd00510871c98d96b1fadce0ec917ecb WatchSource:0}: Error finding container cae2fa77888b0f263d6766157c52f9c4dd00510871c98d96b1fadce0ec917ecb: Status 404 returned error can't find the container with id cae2fa77888b0f263d6766157c52f9c4dd00510871c98d96b1fadce0ec917ecb Sep 08 18:52:44.609060 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.609029 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:52:44.609134 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.609098 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:52:44.609134 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.609128 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:52:44.745555 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.745524 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="433527c6-cda0-4dbd-bb6c-0f2455dbb588" path="/var/lib/kubelet/pods/433527c6-cda0-4dbd-bb6c-0f2455dbb588/volumes" Sep 08 18:52:44.745851 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:44.745838 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cebeb885-8556-444d-9383-61bbf34a55ba" path="/var/lib/kubelet/pods/cebeb885-8556-444d-9383-61bbf34a55ba/volumes" Sep 08 18:52:45.515273 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:45.515238 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" event={"ID":"a25567e8-7d2d-4f0e-9551-a4054118dfd6","Type":"ContainerStarted","Data":"2a618511f9f262754d6a82caf16b3dfb5f70cde50f693ac999ff5db58e982732"} Sep 08 18:52:45.515768 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:45.515311 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" event={"ID":"a25567e8-7d2d-4f0e-9551-a4054118dfd6","Type":"ContainerStarted","Data":"cae2fa77888b0f263d6766157c52f9c4dd00510871c98d96b1fadce0ec917ecb"} Sep 08 18:52:45.536744 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:45.536699 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" podStartSLOduration=2.536684329 podStartE2EDuration="2.536684329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:52:45.535655395 +0000 UTC m=+1101.241543632" watchObservedRunningTime="2026-09-08 18:52:45.536684329 +0000 UTC m=+1101.242572561" Sep 08 18:52:46.326760 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:46.326724 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:46.331551 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:46.331522 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:46.521793 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:46.521751 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:46.522611 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:46.522590 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:52:48.226971 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.226947 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:52:48.261422 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.261331 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-podinfo\") pod \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " Sep 08 18:52:48.261555 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.261427 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg9w7\" (UniqueName: \"kubernetes.io/projected/66b6a2fc-2a2d-41a2-9455-d9a305177d33-kube-api-access-jg9w7\") pod \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " Sep 08 18:52:48.261555 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.261468 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-workload-socket\") pod \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " Sep 08 18:52:48.261555 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.261526 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-credential-socket\") pod \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " Sep 08 18:52:48.261714 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.261567 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-envoy\") pod \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " Sep 08 18:52:48.261714 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.261593 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istiod-ca-cert\") pod \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " Sep 08 18:52:48.261714 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.261616 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-token\") pod \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " Sep 08 18:52:48.261714 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.261640 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-data\") pod \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " Sep 08 18:52:48.261714 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.261676 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-workload-certs\") pod \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\" (UID: \"66b6a2fc-2a2d-41a2-9455-d9a305177d33\") " Sep 08 18:52:48.261974 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.261783 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "66b6a2fc-2a2d-41a2-9455-d9a305177d33" (UID: "66b6a2fc-2a2d-41a2-9455-d9a305177d33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:52:48.262154 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.262044 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "66b6a2fc-2a2d-41a2-9455-d9a305177d33" (UID: "66b6a2fc-2a2d-41a2-9455-d9a305177d33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:52:48.262154 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.262056 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "66b6a2fc-2a2d-41a2-9455-d9a305177d33" (UID: "66b6a2fc-2a2d-41a2-9455-d9a305177d33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:52:48.262154 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.262074 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:52:48.262154 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.262118 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "66b6a2fc-2a2d-41a2-9455-d9a305177d33" (UID: "66b6a2fc-2a2d-41a2-9455-d9a305177d33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:52:48.262440 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.262261 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-data" (OuterVolumeSpecName: "istio-data") pod "66b6a2fc-2a2d-41a2-9455-d9a305177d33" (UID: "66b6a2fc-2a2d-41a2-9455-d9a305177d33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:52:48.263747 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.263722 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b6a2fc-2a2d-41a2-9455-d9a305177d33-kube-api-access-jg9w7" (OuterVolumeSpecName: "kube-api-access-jg9w7") pod "66b6a2fc-2a2d-41a2-9455-d9a305177d33" (UID: "66b6a2fc-2a2d-41a2-9455-d9a305177d33"). InnerVolumeSpecName "kube-api-access-jg9w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:52:48.263927 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.263899 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "66b6a2fc-2a2d-41a2-9455-d9a305177d33" (UID: "66b6a2fc-2a2d-41a2-9455-d9a305177d33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 18:52:48.264153 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.264135 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-token" (OuterVolumeSpecName: "istio-token") pod "66b6a2fc-2a2d-41a2-9455-d9a305177d33" (UID: "66b6a2fc-2a2d-41a2-9455-d9a305177d33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:52:48.264460 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.264409 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "66b6a2fc-2a2d-41a2-9455-d9a305177d33" (UID: "66b6a2fc-2a2d-41a2-9455-d9a305177d33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:52:48.362623 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.362592 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:52:48.362623 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.362617 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:52:48.362623 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.362627 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:52:48.362833 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.362635 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:52:48.362833 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.362645 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:52:48.362833 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.362652 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66b6a2fc-2a2d-41a2-9455-d9a305177d33-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:52:48.362833 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.362660 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66b6a2fc-2a2d-41a2-9455-d9a305177d33-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:52:48.362833 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.362667 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg9w7\" (UniqueName: \"kubernetes.io/projected/66b6a2fc-2a2d-41a2-9455-d9a305177d33-kube-api-access-jg9w7\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:52:48.534625 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.534541 2553 generic.go:356] "Generic (PLEG): container finished" podID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerID="e38616a674afc7a8856cfea2c7004e66979a9235f3aa08e04d2204e4059e8c4f" exitCode=0 Sep 08 18:52:48.534625 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.534606 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" Sep 08 18:52:48.534817 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.534638 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" event={"ID":"66b6a2fc-2a2d-41a2-9455-d9a305177d33","Type":"ContainerDied","Data":"e38616a674afc7a8856cfea2c7004e66979a9235f3aa08e04d2204e4059e8c4f"} Sep 08 18:52:48.534817 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.534697 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx" event={"ID":"66b6a2fc-2a2d-41a2-9455-d9a305177d33","Type":"ContainerDied","Data":"d2947f08d77253c5d54d1716b7acac42446f813eed19f19ac7faf63e36c652bc"} Sep 08 18:52:48.534817 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.534728 2553 scope.go:117] "RemoveContainer" containerID="e38616a674afc7a8856cfea2c7004e66979a9235f3aa08e04d2204e4059e8c4f" Sep 08 18:52:48.549284 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.549257 2553 scope.go:117] "RemoveContainer" containerID="7cf36c47f9ee15979e5511bce18f3eb36cacce60c37f9f4f7e04a4275b311171" Sep 08 18:52:48.557941 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.557916 2553 scope.go:117] "RemoveContainer" containerID="e38616a674afc7a8856cfea2c7004e66979a9235f3aa08e04d2204e4059e8c4f" Sep 08 18:52:48.558263 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:52:48.558236 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38616a674afc7a8856cfea2c7004e66979a9235f3aa08e04d2204e4059e8c4f\": container with ID starting with e38616a674afc7a8856cfea2c7004e66979a9235f3aa08e04d2204e4059e8c4f not found: ID does not exist" containerID="e38616a674afc7a8856cfea2c7004e66979a9235f3aa08e04d2204e4059e8c4f" Sep 08 18:52:48.558476 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.558281 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38616a674afc7a8856cfea2c7004e66979a9235f3aa08e04d2204e4059e8c4f"} err="failed to get container status \"e38616a674afc7a8856cfea2c7004e66979a9235f3aa08e04d2204e4059e8c4f\": rpc error: code = NotFound desc = could not find container \"e38616a674afc7a8856cfea2c7004e66979a9235f3aa08e04d2204e4059e8c4f\": container with ID starting with e38616a674afc7a8856cfea2c7004e66979a9235f3aa08e04d2204e4059e8c4f not found: ID does not exist" Sep 08 18:52:48.558476 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.558300 2553 scope.go:117] "RemoveContainer" containerID="7cf36c47f9ee15979e5511bce18f3eb36cacce60c37f9f4f7e04a4275b311171" Sep 08 18:52:48.558619 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:52:48.558600 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf36c47f9ee15979e5511bce18f3eb36cacce60c37f9f4f7e04a4275b311171\": container with ID starting with 7cf36c47f9ee15979e5511bce18f3eb36cacce60c37f9f4f7e04a4275b311171 not found: ID does not exist" containerID="7cf36c47f9ee15979e5511bce18f3eb36cacce60c37f9f4f7e04a4275b311171" Sep 08 18:52:48.558676 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.558632 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf36c47f9ee15979e5511bce18f3eb36cacce60c37f9f4f7e04a4275b311171"} err="failed to get container status \"7cf36c47f9ee15979e5511bce18f3eb36cacce60c37f9f4f7e04a4275b311171\": rpc error: code = NotFound desc = could not find container \"7cf36c47f9ee15979e5511bce18f3eb36cacce60c37f9f4f7e04a4275b311171\": container with ID starting with 7cf36c47f9ee15979e5511bce18f3eb36cacce60c37f9f4f7e04a4275b311171 not found: ID does not exist" Sep 08 18:52:48.559828 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.559806 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx"] Sep 08 18:52:48.563792 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.563767 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ffda9270-openshift-default-744696f866-6g7qx"] Sep 08 18:52:48.746119 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:52:48.746086 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" path="/var/lib/kubelet/pods/66b6a2fc-2a2d-41a2-9455-d9a305177d33/volumes" Sep 08 18:53:02.563542 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:02.563495 2553 patch_prober.go:28] interesting pod/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 18:53:02.563958 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:02.563593 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" podUID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 18:53:04.622925 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:04.622896 2553 generic.go:356] "Generic (PLEG): container finished" podID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" containerID="2a618511f9f262754d6a82caf16b3dfb5f70cde50f693ac999ff5db58e982732" exitCode=0 Sep 08 18:53:04.623288 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:04.622968 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" event={"ID":"a25567e8-7d2d-4f0e-9551-a4054118dfd6","Type":"ContainerDied","Data":"2a618511f9f262754d6a82caf16b3dfb5f70cde50f693ac999ff5db58e982732"} Sep 08 18:53:04.623349 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:04.623334 2553 scope.go:117] "RemoveContainer" containerID="2a618511f9f262754d6a82caf16b3dfb5f70cde50f693ac999ff5db58e982732" Sep 08 18:53:04.624395 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:04.624351 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:53:04.624451 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:04.624432 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:53:04.624489 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:04.624463 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:53:05.631459 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:05.631415 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" event={"ID":"a25567e8-7d2d-4f0e-9551-a4054118dfd6","Type":"ContainerStarted","Data":"10748c678e0376ff49b4a5e1a3de852591006524269b9cc92a1a232849409c6f"} Sep 08 18:53:06.326044 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:06.326012 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:53:06.326205 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:06.326164 2553 patch_prober.go:28] interesting pod/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 08 18:53:06.326250 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:06.326213 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" podUID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 08 18:53:07.326739 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:07.326702 2553 patch_prober.go:28] interesting pod/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 08 18:53:07.327150 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:07.326757 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" podUID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 08 18:53:08.326888 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:08.326847 2553 patch_prober.go:28] interesting pod/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 08 18:53:08.327452 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:08.326912 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" podUID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 08 18:53:09.326118 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:09.326081 2553 patch_prober.go:28] interesting pod/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 08 18:53:09.326287 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:09.326168 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" podUID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 08 18:53:10.326772 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:10.326734 2553 patch_prober.go:28] interesting pod/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 08 18:53:10.327153 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:10.326813 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" podUID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 08 18:53:11.330060 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:11.330028 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:53:11.330474 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:11.330395 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:53:11.331061 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:11.331043 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:53:12.106606 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:12.106570 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw"] Sep 08 18:53:12.107204 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:12.107181 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerName="istio-proxy" Sep 08 18:53:12.107301 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:12.107207 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerName="istio-proxy" Sep 08 18:53:12.107301 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:12.107221 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerName="istio-proxy" Sep 08 18:53:12.107301 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:12.107231 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerName="istio-proxy" Sep 08 18:53:12.107428 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:12.107339 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerName="istio-proxy" Sep 08 18:53:12.110266 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:12.110247 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw" Sep 08 18:53:12.113243 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:12.113214 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 18:53:12.113390 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:12.113282 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 18:53:12.117838 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:12.117814 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw"] Sep 08 18:53:12.206278 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:12.206243 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c580eab6-4446-44b6-b903-c4bb361107d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw\" (UID: \"c580eab6-4446-44b6-b903-c4bb361107d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw" Sep 08 18:53:12.306996 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:12.306967 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c580eab6-4446-44b6-b903-c4bb361107d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw\" (UID: \"c580eab6-4446-44b6-b903-c4bb361107d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw" Sep 08 18:53:12.307674 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:12.307648 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c580eab6-4446-44b6-b903-c4bb361107d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw\" (UID: \"c580eab6-4446-44b6-b903-c4bb361107d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw" Sep 08 18:53:12.423122 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:12.423093 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw" Sep 08 18:53:12.697800 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:12.697777 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw"] Sep 08 18:53:12.699697 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:53:12.699672 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc580eab6_4446_44b6_b903_c4bb361107d3.slice/crio-ebd1e7411fe3eb8bf0420a2176f432388ce054c6d7018426c8dc6f86811c6026 WatchSource:0}: Error finding container ebd1e7411fe3eb8bf0420a2176f432388ce054c6d7018426c8dc6f86811c6026: Status 404 returned error can't find the container with id ebd1e7411fe3eb8bf0420a2176f432388ce054c6d7018426c8dc6f86811c6026 Sep 08 18:53:13.689188 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:13.689095 2553 generic.go:356] "Generic (PLEG): container finished" podID="c580eab6-4446-44b6-b903-c4bb361107d3" containerID="b7624a953a815cc6b3f5ac5c761c5da408f5ccefba1f014edd931e1aebb0574d" exitCode=0 Sep 08 18:53:13.689188 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:13.689173 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw" event={"ID":"c580eab6-4446-44b6-b903-c4bb361107d3","Type":"ContainerDied","Data":"b7624a953a815cc6b3f5ac5c761c5da408f5ccefba1f014edd931e1aebb0574d"} Sep 08 18:53:13.689669 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:13.689232 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw" event={"ID":"c580eab6-4446-44b6-b903-c4bb361107d3","Type":"ContainerStarted","Data":"ebd1e7411fe3eb8bf0420a2176f432388ce054c6d7018426c8dc6f86811c6026"} Sep 08 18:53:14.835940 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:14.835913 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw" Sep 08 18:53:14.931228 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:14.931196 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c580eab6-4446-44b6-b903-c4bb361107d3-maas-api-service-ca\") pod \"c580eab6-4446-44b6-b903-c4bb361107d3\" (UID: \"c580eab6-4446-44b6-b903-c4bb361107d3\") " Sep 08 18:53:14.931528 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:14.931506 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c580eab6-4446-44b6-b903-c4bb361107d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c580eab6-4446-44b6-b903-c4bb361107d3" (UID: "c580eab6-4446-44b6-b903-c4bb361107d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:53:15.032775 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:15.032704 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c580eab6-4446-44b6-b903-c4bb361107d3-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:53:15.701295 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:15.701266 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw" Sep 08 18:53:15.701295 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:15.701291 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw" event={"ID":"c580eab6-4446-44b6-b903-c4bb361107d3","Type":"ContainerDied","Data":"ebd1e7411fe3eb8bf0420a2176f432388ce054c6d7018426c8dc6f86811c6026"} Sep 08 18:53:15.701533 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:15.701337 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebd1e7411fe3eb8bf0420a2176f432388ce054c6d7018426c8dc6f86811c6026" Sep 08 18:53:18.702844 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:18.702803 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw"] Sep 08 18:53:18.705509 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:18.705486 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-62eb1773-g95cw"] Sep 08 18:53:18.744798 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:18.744768 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c580eab6-4446-44b6-b903-c4bb361107d3" path="/var/lib/kubelet/pods/c580eab6-4446-44b6-b903-c4bb361107d3/volumes" Sep 08 18:53:18.855891 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:18.855857 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh"] Sep 08 18:53:18.856085 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:18.856061 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" podUID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" containerName="istio-proxy" containerID="cri-o://10748c678e0376ff49b4a5e1a3de852591006524269b9cc92a1a232849409c6f" gracePeriod=30 Sep 08 18:53:19.642465 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.642430 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss"] Sep 08 18:53:19.642993 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.642963 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c580eab6-4446-44b6-b903-c4bb361107d3" containerName="revoke-keys" Sep 08 18:53:19.642993 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.642991 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c580eab6-4446-44b6-b903-c4bb361107d3" containerName="revoke-keys" Sep 08 18:53:19.643208 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.643082 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c580eab6-4446-44b6-b903-c4bb361107d3" containerName="revoke-keys" Sep 08 18:53:19.643208 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.643104 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="66b6a2fc-2a2d-41a2-9455-d9a305177d33" containerName="istio-proxy" Sep 08 18:53:19.645767 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.645741 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.648475 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.648453 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a6830f31-openshift-default-dockercfg-cd7kq\"" Sep 08 18:53:19.659014 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.658987 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss"] Sep 08 18:53:19.779338 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.779299 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-podinfo\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.779748 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.779350 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-credential-socket\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.779748 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.779429 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-token\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.779748 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.779522 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-data\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.779748 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.779596 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-workload-certs\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.779748 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.779620 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfpr2\" (UniqueName: \"kubernetes.io/projected/a7d224cc-495b-47ca-a003-51cfc8fc2722-kube-api-access-tfpr2\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.779748 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.779639 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-envoy\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.779748 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.779655 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d224cc-495b-47ca-a003-51cfc8fc2722-istiod-ca-cert\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.779748 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.779700 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-workload-socket\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.880721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.880686 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-envoy\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.880721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.880721 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d224cc-495b-47ca-a003-51cfc8fc2722-istiod-ca-cert\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.880970 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.880741 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-workload-socket\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.880970 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.880795 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-podinfo\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.881086 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.881006 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-credential-socket\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.881086 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.881053 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-token\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.881214 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.881188 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-workload-socket\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.881316 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.881296 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-data\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.881446 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.881312 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-credential-socket\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.881602 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.881504 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d224cc-495b-47ca-a003-51cfc8fc2722-istiod-ca-cert\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.881602 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.881532 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-workload-certs\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.881602 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.881559 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-data\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.881602 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.881583 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfpr2\" (UniqueName: \"kubernetes.io/projected/a7d224cc-495b-47ca-a003-51cfc8fc2722-kube-api-access-tfpr2\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.881856 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.881836 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-workload-certs\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.883205 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.883187 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-envoy\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.883551 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.883532 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-podinfo\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.888835 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.888807 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-token\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.888916 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.888894 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfpr2\" (UniqueName: \"kubernetes.io/projected/a7d224cc-495b-47ca-a003-51cfc8fc2722-kube-api-access-tfpr2\") pod \"e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:19.957471 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:19.957418 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:20.232582 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:20.232551 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss"] Sep 08 18:53:20.233541 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:53:20.233510 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d224cc_495b_47ca_a003_51cfc8fc2722.slice/crio-9b3ce8d7215ebbd741a3a2d9458dc24c61afa8c5537eaf3bf89dc5bc09ea99b8 WatchSource:0}: Error finding container 9b3ce8d7215ebbd741a3a2d9458dc24c61afa8c5537eaf3bf89dc5bc09ea99b8: Status 404 returned error can't find the container with id 9b3ce8d7215ebbd741a3a2d9458dc24c61afa8c5537eaf3bf89dc5bc09ea99b8 Sep 08 18:53:20.235901 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:20.235866 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:53:20.235975 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:20.235937 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:53:20.235975 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:20.235965 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:53:20.729832 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:20.729799 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" event={"ID":"a7d224cc-495b-47ca-a003-51cfc8fc2722","Type":"ContainerStarted","Data":"13056ac285a1690c6941b45d4a4b21edcfcbd868dd13e6763cb3719ebb28c316"} Sep 08 18:53:20.730001 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:20.729856 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" event={"ID":"a7d224cc-495b-47ca-a003-51cfc8fc2722","Type":"ContainerStarted","Data":"9b3ce8d7215ebbd741a3a2d9458dc24c61afa8c5537eaf3bf89dc5bc09ea99b8"} Sep 08 18:53:20.753406 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:20.753339 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" podStartSLOduration=1.7533225030000001 podStartE2EDuration="1.753322503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:53:20.75089185 +0000 UTC m=+1136.456780081" watchObservedRunningTime="2026-09-08 18:53:20.753322503 +0000 UTC m=+1136.459210810" Sep 08 18:53:20.957967 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:20.957936 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:21.962734 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:21.962708 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:22.745036 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:22.745007 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:22.745201 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:22.745072 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:24.115034 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.115004 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:53:24.122458 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.122435 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-token\") pod \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " Sep 08 18:53:24.122571 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.122517 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbwhc\" (UniqueName: \"kubernetes.io/projected/a25567e8-7d2d-4f0e-9551-a4054118dfd6-kube-api-access-rbwhc\") pod \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " Sep 08 18:53:24.122571 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.122552 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-workload-socket\") pod \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " Sep 08 18:53:24.122749 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.122619 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-envoy\") pod \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " Sep 08 18:53:24.122749 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.122693 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-workload-certs\") pod \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " Sep 08 18:53:24.122749 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.122731 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istiod-ca-cert\") pod \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " Sep 08 18:53:24.122903 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.122777 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-data\") pod \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " Sep 08 18:53:24.122903 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.122802 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-podinfo\") pod \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " Sep 08 18:53:24.122903 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.122827 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-credential-socket\") pod \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\" (UID: \"a25567e8-7d2d-4f0e-9551-a4054118dfd6\") " Sep 08 18:53:24.122903 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.122856 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a25567e8-7d2d-4f0e-9551-a4054118dfd6" (UID: "a25567e8-7d2d-4f0e-9551-a4054118dfd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:53:24.123096 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.122965 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a25567e8-7d2d-4f0e-9551-a4054118dfd6" (UID: "a25567e8-7d2d-4f0e-9551-a4054118dfd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:53:24.123096 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.123083 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-data" (OuterVolumeSpecName: "istio-data") pod "a25567e8-7d2d-4f0e-9551-a4054118dfd6" (UID: "a25567e8-7d2d-4f0e-9551-a4054118dfd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:53:24.123209 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.123119 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a25567e8-7d2d-4f0e-9551-a4054118dfd6" (UID: "a25567e8-7d2d-4f0e-9551-a4054118dfd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:53:24.123209 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.123200 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:53:24.123314 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.123220 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:53:24.123314 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.123257 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:53:24.123419 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.123310 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a25567e8-7d2d-4f0e-9551-a4054118dfd6" (UID: "a25567e8-7d2d-4f0e-9551-a4054118dfd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:53:24.124714 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.124691 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a25567e8-7d2d-4f0e-9551-a4054118dfd6" (UID: "a25567e8-7d2d-4f0e-9551-a4054118dfd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:53:24.124813 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.124752 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25567e8-7d2d-4f0e-9551-a4054118dfd6-kube-api-access-rbwhc" (OuterVolumeSpecName: "kube-api-access-rbwhc") pod "a25567e8-7d2d-4f0e-9551-a4054118dfd6" (UID: "a25567e8-7d2d-4f0e-9551-a4054118dfd6"). InnerVolumeSpecName "kube-api-access-rbwhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:53:24.124963 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.124945 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-token" (OuterVolumeSpecName: "istio-token") pod "a25567e8-7d2d-4f0e-9551-a4054118dfd6" (UID: "a25567e8-7d2d-4f0e-9551-a4054118dfd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:53:24.125076 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.125061 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a25567e8-7d2d-4f0e-9551-a4054118dfd6" (UID: "a25567e8-7d2d-4f0e-9551-a4054118dfd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 18:53:24.224476 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.224442 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbwhc\" (UniqueName: \"kubernetes.io/projected/a25567e8-7d2d-4f0e-9551-a4054118dfd6-kube-api-access-rbwhc\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:53:24.224476 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.224467 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:53:24.224476 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.224478 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:53:24.224689 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.224486 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:53:24.224689 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.224494 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25567e8-7d2d-4f0e-9551-a4054118dfd6-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:53:24.224689 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.224501 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25567e8-7d2d-4f0e-9551-a4054118dfd6-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:53:24.764305 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.764272 2553 generic.go:356] "Generic (PLEG): container finished" podID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" containerID="10748c678e0376ff49b4a5e1a3de852591006524269b9cc92a1a232849409c6f" exitCode=0 Sep 08 18:53:24.764462 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.764317 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" Sep 08 18:53:24.764462 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.764345 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" event={"ID":"a25567e8-7d2d-4f0e-9551-a4054118dfd6","Type":"ContainerDied","Data":"10748c678e0376ff49b4a5e1a3de852591006524269b9cc92a1a232849409c6f"} Sep 08 18:53:24.764462 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.764410 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh" event={"ID":"a25567e8-7d2d-4f0e-9551-a4054118dfd6","Type":"ContainerDied","Data":"cae2fa77888b0f263d6766157c52f9c4dd00510871c98d96b1fadce0ec917ecb"} Sep 08 18:53:24.764462 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.764433 2553 scope.go:117] "RemoveContainer" containerID="10748c678e0376ff49b4a5e1a3de852591006524269b9cc92a1a232849409c6f" Sep 08 18:53:24.773660 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.773644 2553 scope.go:117] "RemoveContainer" containerID="2a618511f9f262754d6a82caf16b3dfb5f70cde50f693ac999ff5db58e982732" Sep 08 18:53:24.782142 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.782123 2553 scope.go:117] "RemoveContainer" containerID="10748c678e0376ff49b4a5e1a3de852591006524269b9cc92a1a232849409c6f" Sep 08 18:53:24.782436 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:53:24.782418 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10748c678e0376ff49b4a5e1a3de852591006524269b9cc92a1a232849409c6f\": container with ID starting with 10748c678e0376ff49b4a5e1a3de852591006524269b9cc92a1a232849409c6f not found: ID does not exist" containerID="10748c678e0376ff49b4a5e1a3de852591006524269b9cc92a1a232849409c6f" Sep 08 18:53:24.782527 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.782443 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10748c678e0376ff49b4a5e1a3de852591006524269b9cc92a1a232849409c6f"} err="failed to get container status \"10748c678e0376ff49b4a5e1a3de852591006524269b9cc92a1a232849409c6f\": rpc error: code = NotFound desc = could not find container \"10748c678e0376ff49b4a5e1a3de852591006524269b9cc92a1a232849409c6f\": container with ID starting with 10748c678e0376ff49b4a5e1a3de852591006524269b9cc92a1a232849409c6f not found: ID does not exist" Sep 08 18:53:24.782527 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.782461 2553 scope.go:117] "RemoveContainer" containerID="2a618511f9f262754d6a82caf16b3dfb5f70cde50f693ac999ff5db58e982732" Sep 08 18:53:24.782727 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:53:24.782704 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a618511f9f262754d6a82caf16b3dfb5f70cde50f693ac999ff5db58e982732\": container with ID starting with 2a618511f9f262754d6a82caf16b3dfb5f70cde50f693ac999ff5db58e982732 not found: ID does not exist" containerID="2a618511f9f262754d6a82caf16b3dfb5f70cde50f693ac999ff5db58e982732" Sep 08 18:53:24.782787 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.782735 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a618511f9f262754d6a82caf16b3dfb5f70cde50f693ac999ff5db58e982732"} err="failed to get container status \"2a618511f9f262754d6a82caf16b3dfb5f70cde50f693ac999ff5db58e982732\": rpc error: code = NotFound desc = could not find container \"2a618511f9f262754d6a82caf16b3dfb5f70cde50f693ac999ff5db58e982732\": container with ID starting with 2a618511f9f262754d6a82caf16b3dfb5f70cde50f693ac999ff5db58e982732 not found: ID does not exist" Sep 08 18:53:24.783489 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.783469 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh"] Sep 08 18:53:24.787305 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:24.787279 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-62eb1773-gw-openshift-default-76fc489df6-dtfjh"] Sep 08 18:53:26.745403 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:26.745346 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" path="/var/lib/kubelet/pods/a25567e8-7d2d-4f0e-9551-a4054118dfd6/volumes" Sep 08 18:53:37.919305 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:37.919269 2553 patch_prober.go:28] interesting pod/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:53790->10.134.0.55:15021: read: connection reset by peer" start-of-body= Sep 08 18:53:37.919750 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:37.919354 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" podUID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:53790->10.134.0.55:15021: read: connection reset by peer" Sep 08 18:53:38.840852 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:38.840820 2553 generic.go:356] "Generic (PLEG): container finished" podID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerID="13056ac285a1690c6941b45d4a4b21edcfcbd868dd13e6763cb3719ebb28c316" exitCode=0 Sep 08 18:53:38.841019 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:38.840895 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" event={"ID":"a7d224cc-495b-47ca-a003-51cfc8fc2722","Type":"ContainerDied","Data":"13056ac285a1690c6941b45d4a4b21edcfcbd868dd13e6763cb3719ebb28c316"} Sep 08 18:53:38.841291 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:38.841278 2553 scope.go:117] "RemoveContainer" containerID="13056ac285a1690c6941b45d4a4b21edcfcbd868dd13e6763cb3719ebb28c316" Sep 08 18:53:38.842353 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:38.842326 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:53:38.842421 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:38.842412 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:53:38.842458 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:38.842446 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:53:39.847863 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:39.847824 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" event={"ID":"a7d224cc-495b-47ca-a003-51cfc8fc2722","Type":"ContainerStarted","Data":"e973e00be811ba24c9b1215beeab1f3f1464154f9a75790263d1cc997c8fa84c"} Sep 08 18:53:39.958055 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:39.958021 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:39.958210 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:39.958184 2553 patch_prober.go:28] interesting pod/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 08 18:53:39.958254 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:39.958233 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" podUID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 08 18:53:40.958779 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:40.958743 2553 patch_prober.go:28] interesting pod/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 08 18:53:40.959170 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:40.958802 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" podUID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 08 18:53:41.957826 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:41.957787 2553 patch_prober.go:28] interesting pod/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 08 18:53:41.958001 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:41.957845 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" podUID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 08 18:53:42.958309 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:42.958280 2553 patch_prober.go:28] interesting pod/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 08 18:53:42.958698 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:42.958335 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" podUID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 08 18:53:43.958500 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:43.958469 2553 patch_prober.go:28] interesting pod/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 08 18:53:43.958874 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:43.958521 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" podUID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 08 18:53:44.957776 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:44.957741 2553 patch_prober.go:28] interesting pod/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 08 18:53:44.958096 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:44.957815 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" podUID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 08 18:53:45.961723 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:45.961691 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:45.962126 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:45.962091 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:45.962863 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:45.962841 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:53:47.954883 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:47.954800 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br"] Sep 08 18:53:47.955400 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:47.955380 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" containerName="istio-proxy" Sep 08 18:53:47.955492 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:47.955402 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" containerName="istio-proxy" Sep 08 18:53:47.955492 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:47.955446 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" containerName="istio-proxy" Sep 08 18:53:47.955492 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:47.955456 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" containerName="istio-proxy" Sep 08 18:53:47.955648 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:47.955559 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" containerName="istio-proxy" Sep 08 18:53:47.955648 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:47.955570 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25567e8-7d2d-4f0e-9551-a4054118dfd6" containerName="istio-proxy" Sep 08 18:53:47.957712 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:47.957691 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br" Sep 08 18:53:47.960068 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:47.960050 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 18:53:47.960167 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:47.960111 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 18:53:47.965075 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:47.965051 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br"] Sep 08 18:53:48.046491 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:48.046453 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd73ee3-ad7b-4949-887a-800a2bb93114-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a6830f31-856br\" (UID: \"cbd73ee3-ad7b-4949-887a-800a2bb93114\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br" Sep 08 18:53:48.148128 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:48.148093 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd73ee3-ad7b-4949-887a-800a2bb93114-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a6830f31-856br\" (UID: \"cbd73ee3-ad7b-4949-887a-800a2bb93114\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br" Sep 08 18:53:48.148721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:48.148699 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd73ee3-ad7b-4949-887a-800a2bb93114-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a6830f31-856br\" (UID: \"cbd73ee3-ad7b-4949-887a-800a2bb93114\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br" Sep 08 18:53:48.270418 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:48.270301 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br" Sep 08 18:53:48.539387 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:48.539345 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br"] Sep 08 18:53:48.541804 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:53:48.541777 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd73ee3_ad7b_4949_887a_800a2bb93114.slice/crio-75a634efde1eab2e4baf807f1761de181840bf8c641e349d82b8763243b3565f WatchSource:0}: Error finding container 75a634efde1eab2e4baf807f1761de181840bf8c641e349d82b8763243b3565f: Status 404 returned error can't find the container with id 75a634efde1eab2e4baf807f1761de181840bf8c641e349d82b8763243b3565f Sep 08 18:53:48.899154 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:48.899120 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br" event={"ID":"cbd73ee3-ad7b-4949-887a-800a2bb93114","Type":"ContainerStarted","Data":"a9d9123ce650e22608da88b24874f0095bfcc32a8b5277b7e2f4b3d87d218af6"} Sep 08 18:53:48.899314 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:48.899177 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br" event={"ID":"cbd73ee3-ad7b-4949-887a-800a2bb93114","Type":"ContainerStarted","Data":"75a634efde1eab2e4baf807f1761de181840bf8c641e349d82b8763243b3565f"} Sep 08 18:53:48.914414 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:48.914341 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br" podStartSLOduration=1.914325287 podStartE2EDuration="1.914325287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:53:48.914242664 +0000 UTC m=+1164.620130895" watchObservedRunningTime="2026-09-08 18:53:48.914325287 +0000 UTC m=+1164.620213518" Sep 08 18:53:49.906604 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:49.906568 2553 generic.go:356] "Generic (PLEG): container finished" podID="cbd73ee3-ad7b-4949-887a-800a2bb93114" containerID="a9d9123ce650e22608da88b24874f0095bfcc32a8b5277b7e2f4b3d87d218af6" exitCode=0 Sep 08 18:53:49.906988 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:49.906665 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br" event={"ID":"cbd73ee3-ad7b-4949-887a-800a2bb93114","Type":"ContainerDied","Data":"a9d9123ce650e22608da88b24874f0095bfcc32a8b5277b7e2f4b3d87d218af6"} Sep 08 18:53:51.045158 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:51.045133 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br" Sep 08 18:53:51.176584 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:51.176504 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd73ee3-ad7b-4949-887a-800a2bb93114-maas-api-service-ca\") pod \"cbd73ee3-ad7b-4949-887a-800a2bb93114\" (UID: \"cbd73ee3-ad7b-4949-887a-800a2bb93114\") " Sep 08 18:53:51.176845 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:51.176823 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd73ee3-ad7b-4949-887a-800a2bb93114-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cbd73ee3-ad7b-4949-887a-800a2bb93114" (UID: "cbd73ee3-ad7b-4949-887a-800a2bb93114"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:53:51.277669 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:51.277642 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd73ee3-ad7b-4949-887a-800a2bb93114-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:53:51.919234 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:51.919198 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br" event={"ID":"cbd73ee3-ad7b-4949-887a-800a2bb93114","Type":"ContainerDied","Data":"75a634efde1eab2e4baf807f1761de181840bf8c641e349d82b8763243b3565f"} Sep 08 18:53:51.919234 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:51.919231 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br" Sep 08 18:53:51.919468 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:51.919248 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75a634efde1eab2e4baf807f1761de181840bf8c641e349d82b8763243b3565f" Sep 08 18:53:53.972177 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:53.972138 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br"] Sep 08 18:53:53.974769 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:53.974742 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6830f31-856br"] Sep 08 18:53:54.745021 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:54.744988 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbd73ee3-ad7b-4949-887a-800a2bb93114" path="/var/lib/kubelet/pods/cbd73ee3-ad7b-4949-887a-800a2bb93114/volumes" Sep 08 18:53:55.205273 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:55.205234 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss"] Sep 08 18:53:55.205767 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:55.205501 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" podUID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerName="istio-proxy" containerID="cri-o://e973e00be811ba24c9b1215beeab1f3f1464154f9a75790263d1cc997c8fa84c" gracePeriod=30 Sep 08 18:53:56.031122 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.031081 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm"] Sep 08 18:53:56.031679 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.031654 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbd73ee3-ad7b-4949-887a-800a2bb93114" containerName="revoke-keys" Sep 08 18:53:56.031679 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.031677 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbd73ee3-ad7b-4949-887a-800a2bb93114" containerName="revoke-keys" Sep 08 18:53:56.031845 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.031775 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbd73ee3-ad7b-4949-887a-800a2bb93114" containerName="revoke-keys" Sep 08 18:53:56.033982 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.033960 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.037323 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.037300 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-6956efcb-openshift-default-dockercfg-mslf8\"" Sep 08 18:53:56.051967 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.051940 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm"] Sep 08 18:53:56.121391 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.121342 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-token\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.121566 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.121406 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-data\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.121566 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.121484 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-credential-socket\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.121566 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.121527 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-workload-certs\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.121566 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.121564 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsd4c\" (UniqueName: \"kubernetes.io/projected/40ad8f7d-0594-4492-841f-966d8a3b9994-kube-api-access-bsd4c\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.121749 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.121600 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-workload-socket\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.121749 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.121684 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-envoy\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.121749 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.121719 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-podinfo\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.122115 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.121770 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ad8f7d-0594-4492-841f-966d8a3b9994-istiod-ca-cert\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.223158 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.223123 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-token\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.223626 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.223171 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-data\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.223626 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.223195 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-credential-socket\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.223626 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.223306 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-workload-certs\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.223626 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.223345 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsd4c\" (UniqueName: \"kubernetes.io/projected/40ad8f7d-0594-4492-841f-966d8a3b9994-kube-api-access-bsd4c\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.223626 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.223434 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-workload-socket\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.223626 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.223502 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-envoy\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.223626 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.223529 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-credential-socket\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.223626 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.223546 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-podinfo\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.223626 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.223591 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-data\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.223626 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.223592 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ad8f7d-0594-4492-841f-966d8a3b9994-istiod-ca-cert\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.223996 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.223668 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-workload-certs\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.223996 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.223808 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-workload-socket\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.224076 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.224057 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ad8f7d-0594-4492-841f-966d8a3b9994-istiod-ca-cert\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.225705 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.225681 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-envoy\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.225905 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.225885 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-podinfo\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.232915 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.232893 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-token\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.233719 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.233696 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsd4c\" (UniqueName: \"kubernetes.io/projected/40ad8f7d-0594-4492-841f-966d8a3b9994-kube-api-access-bsd4c\") pod \"e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.345315 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.345254 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:56.624374 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.624332 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm"] Sep 08 18:53:56.625404 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:53:56.625357 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ad8f7d_0594_4492_841f_966d8a3b9994.slice/crio-75c07183d016abe368d7e9248a2280237007a3f25c42215907cb89e1a3834c88 WatchSource:0}: Error finding container 75c07183d016abe368d7e9248a2280237007a3f25c42215907cb89e1a3834c88: Status 404 returned error can't find the container with id 75c07183d016abe368d7e9248a2280237007a3f25c42215907cb89e1a3834c88 Sep 08 18:53:56.628063 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.628025 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:53:56.628181 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.628112 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:53:56.628181 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.628167 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:53:56.949155 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.949074 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" event={"ID":"40ad8f7d-0594-4492-841f-966d8a3b9994","Type":"ContainerStarted","Data":"04dc394c5eceaa5b5970d471233656c14ba577ac7e8a298d8501b8f8d0a360e7"} Sep 08 18:53:56.949155 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.949137 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" event={"ID":"40ad8f7d-0594-4492-841f-966d8a3b9994","Type":"ContainerStarted","Data":"75c07183d016abe368d7e9248a2280237007a3f25c42215907cb89e1a3834c88"} Sep 08 18:53:56.968075 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:56.968024 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" podStartSLOduration=0.968002651 podStartE2EDuration="968.002651ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:53:56.967117632 +0000 UTC m=+1172.673005866" watchObservedRunningTime="2026-09-08 18:53:56.968002651 +0000 UTC m=+1172.673890881" Sep 08 18:53:57.345963 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:57.345875 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:57.351303 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:57.351276 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:57.955409 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:57.955358 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:53:57.956189 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:53:57.956171 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:54:00.471218 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.471162 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:54:00.565440 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.565411 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-credential-socket\") pod \"a7d224cc-495b-47ca-a003-51cfc8fc2722\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " Sep 08 18:54:00.565618 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.565453 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfpr2\" (UniqueName: \"kubernetes.io/projected/a7d224cc-495b-47ca-a003-51cfc8fc2722-kube-api-access-tfpr2\") pod \"a7d224cc-495b-47ca-a003-51cfc8fc2722\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " Sep 08 18:54:00.565618 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.565473 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-workload-socket\") pod \"a7d224cc-495b-47ca-a003-51cfc8fc2722\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " Sep 08 18:54:00.565618 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.565506 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-podinfo\") pod \"a7d224cc-495b-47ca-a003-51cfc8fc2722\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " Sep 08 18:54:00.565618 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.565529 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-workload-certs\") pod \"a7d224cc-495b-47ca-a003-51cfc8fc2722\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " Sep 08 18:54:00.565618 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.565569 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-data\") pod \"a7d224cc-495b-47ca-a003-51cfc8fc2722\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " Sep 08 18:54:00.565618 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.565585 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d224cc-495b-47ca-a003-51cfc8fc2722-istiod-ca-cert\") pod \"a7d224cc-495b-47ca-a003-51cfc8fc2722\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " Sep 08 18:54:00.565933 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.565646 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-envoy\") pod \"a7d224cc-495b-47ca-a003-51cfc8fc2722\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " Sep 08 18:54:00.565933 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.565679 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-token\") pod \"a7d224cc-495b-47ca-a003-51cfc8fc2722\" (UID: \"a7d224cc-495b-47ca-a003-51cfc8fc2722\") " Sep 08 18:54:00.565933 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.565723 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7d224cc-495b-47ca-a003-51cfc8fc2722" (UID: "a7d224cc-495b-47ca-a003-51cfc8fc2722"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:54:00.565933 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.565902 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7d224cc-495b-47ca-a003-51cfc8fc2722" (UID: "a7d224cc-495b-47ca-a003-51cfc8fc2722"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:54:00.565933 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.565882 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7d224cc-495b-47ca-a003-51cfc8fc2722" (UID: "a7d224cc-495b-47ca-a003-51cfc8fc2722"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:54:00.565933 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.565925 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7d224cc-495b-47ca-a003-51cfc8fc2722" (UID: "a7d224cc-495b-47ca-a003-51cfc8fc2722"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:54:00.566210 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.566022 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:00.566210 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.566042 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:00.566210 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.566056 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:00.566210 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.566070 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:00.566210 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.566077 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d224cc-495b-47ca-a003-51cfc8fc2722-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7d224cc-495b-47ca-a003-51cfc8fc2722" (UID: "a7d224cc-495b-47ca-a003-51cfc8fc2722"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:54:00.567734 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.567703 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d224cc-495b-47ca-a003-51cfc8fc2722-kube-api-access-tfpr2" (OuterVolumeSpecName: "kube-api-access-tfpr2") pod "a7d224cc-495b-47ca-a003-51cfc8fc2722" (UID: "a7d224cc-495b-47ca-a003-51cfc8fc2722"). InnerVolumeSpecName "kube-api-access-tfpr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:54:00.567872 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.567828 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7d224cc-495b-47ca-a003-51cfc8fc2722" (UID: "a7d224cc-495b-47ca-a003-51cfc8fc2722"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 18:54:00.567959 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.567937 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7d224cc-495b-47ca-a003-51cfc8fc2722" (UID: "a7d224cc-495b-47ca-a003-51cfc8fc2722"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:54:00.568078 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.568054 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7d224cc-495b-47ca-a003-51cfc8fc2722" (UID: "a7d224cc-495b-47ca-a003-51cfc8fc2722"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:54:00.666767 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.666736 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:00.666767 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.666760 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:00.666767 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.666771 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfpr2\" (UniqueName: \"kubernetes.io/projected/a7d224cc-495b-47ca-a003-51cfc8fc2722-kube-api-access-tfpr2\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:00.666975 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.666782 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d224cc-495b-47ca-a003-51cfc8fc2722-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:00.666975 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.666790 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d224cc-495b-47ca-a003-51cfc8fc2722-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:00.973518 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.973478 2553 generic.go:356] "Generic (PLEG): container finished" podID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerID="e973e00be811ba24c9b1215beeab1f3f1464154f9a75790263d1cc997c8fa84c" exitCode=0 Sep 08 18:54:00.973658 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.973525 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" Sep 08 18:54:00.973658 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.973554 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" event={"ID":"a7d224cc-495b-47ca-a003-51cfc8fc2722","Type":"ContainerDied","Data":"e973e00be811ba24c9b1215beeab1f3f1464154f9a75790263d1cc997c8fa84c"} Sep 08 18:54:00.973658 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.973604 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss" event={"ID":"a7d224cc-495b-47ca-a003-51cfc8fc2722","Type":"ContainerDied","Data":"9b3ce8d7215ebbd741a3a2d9458dc24c61afa8c5537eaf3bf89dc5bc09ea99b8"} Sep 08 18:54:00.973658 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.973626 2553 scope.go:117] "RemoveContainer" containerID="e973e00be811ba24c9b1215beeab1f3f1464154f9a75790263d1cc997c8fa84c" Sep 08 18:54:00.983317 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.983299 2553 scope.go:117] "RemoveContainer" containerID="13056ac285a1690c6941b45d4a4b21edcfcbd868dd13e6763cb3719ebb28c316" Sep 08 18:54:00.991528 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.991509 2553 scope.go:117] "RemoveContainer" containerID="e973e00be811ba24c9b1215beeab1f3f1464154f9a75790263d1cc997c8fa84c" Sep 08 18:54:00.991795 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:54:00.991776 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e973e00be811ba24c9b1215beeab1f3f1464154f9a75790263d1cc997c8fa84c\": container with ID starting with e973e00be811ba24c9b1215beeab1f3f1464154f9a75790263d1cc997c8fa84c not found: ID does not exist" containerID="e973e00be811ba24c9b1215beeab1f3f1464154f9a75790263d1cc997c8fa84c" Sep 08 18:54:00.991855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.991804 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e973e00be811ba24c9b1215beeab1f3f1464154f9a75790263d1cc997c8fa84c"} err="failed to get container status \"e973e00be811ba24c9b1215beeab1f3f1464154f9a75790263d1cc997c8fa84c\": rpc error: code = NotFound desc = could not find container \"e973e00be811ba24c9b1215beeab1f3f1464154f9a75790263d1cc997c8fa84c\": container with ID starting with e973e00be811ba24c9b1215beeab1f3f1464154f9a75790263d1cc997c8fa84c not found: ID does not exist" Sep 08 18:54:00.991855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.991819 2553 scope.go:117] "RemoveContainer" containerID="13056ac285a1690c6941b45d4a4b21edcfcbd868dd13e6763cb3719ebb28c316" Sep 08 18:54:00.992078 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:54:00.992060 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13056ac285a1690c6941b45d4a4b21edcfcbd868dd13e6763cb3719ebb28c316\": container with ID starting with 13056ac285a1690c6941b45d4a4b21edcfcbd868dd13e6763cb3719ebb28c316 not found: ID does not exist" containerID="13056ac285a1690c6941b45d4a4b21edcfcbd868dd13e6763cb3719ebb28c316" Sep 08 18:54:00.992133 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.992084 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13056ac285a1690c6941b45d4a4b21edcfcbd868dd13e6763cb3719ebb28c316"} err="failed to get container status \"13056ac285a1690c6941b45d4a4b21edcfcbd868dd13e6763cb3719ebb28c316\": rpc error: code = NotFound desc = could not find container \"13056ac285a1690c6941b45d4a4b21edcfcbd868dd13e6763cb3719ebb28c316\": container with ID starting with 13056ac285a1690c6941b45d4a4b21edcfcbd868dd13e6763cb3719ebb28c316 not found: ID does not exist" Sep 08 18:54:00.996825 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:00.996804 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss"] Sep 08 18:54:01.004884 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:01.004830 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a6830f31-openshift-default-569c658bd4-p57ss"] Sep 08 18:54:02.745468 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:02.745436 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7d224cc-495b-47ca-a003-51cfc8fc2722" path="/var/lib/kubelet/pods/a7d224cc-495b-47ca-a003-51cfc8fc2722/volumes" Sep 08 18:54:17.465086 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:17.465045 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw"] Sep 08 18:54:17.465697 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:17.465676 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerName="istio-proxy" Sep 08 18:54:17.465743 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:17.465702 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerName="istio-proxy" Sep 08 18:54:17.465856 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:17.465842 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerName="istio-proxy" Sep 08 18:54:17.469285 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:17.469266 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw" Sep 08 18:54:17.472188 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:17.472161 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 18:54:17.472509 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:17.472485 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 18:54:17.476849 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:17.476826 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw"] Sep 08 18:54:17.519454 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:17.519402 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f55817b-93a2-43a3-807c-add0c90ff8d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-235f65-zp5bw\" (UID: \"0f55817b-93a2-43a3-807c-add0c90ff8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw" Sep 08 18:54:17.620426 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:17.620384 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f55817b-93a2-43a3-807c-add0c90ff8d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-235f65-zp5bw\" (UID: \"0f55817b-93a2-43a3-807c-add0c90ff8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw" Sep 08 18:54:17.621109 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:17.621081 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f55817b-93a2-43a3-807c-add0c90ff8d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-235f65-zp5bw\" (UID: \"0f55817b-93a2-43a3-807c-add0c90ff8d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw" Sep 08 18:54:17.782572 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:17.782487 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw" Sep 08 18:54:18.057185 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:18.057149 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw"] Sep 08 18:54:18.058462 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:54:18.058433 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f55817b_93a2_43a3_807c_add0c90ff8d0.slice/crio-d5f75c14430a9e759914adc837c3d54e7ac562d78c167c3b0c00afa523a5548f WatchSource:0}: Error finding container d5f75c14430a9e759914adc837c3d54e7ac562d78c167c3b0c00afa523a5548f: Status 404 returned error can't find the container with id d5f75c14430a9e759914adc837c3d54e7ac562d78c167c3b0c00afa523a5548f Sep 08 18:54:18.083117 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:18.083089 2553 generic.go:356] "Generic (PLEG): container finished" podID="40ad8f7d-0594-4492-841f-966d8a3b9994" containerID="04dc394c5eceaa5b5970d471233656c14ba577ac7e8a298d8501b8f8d0a360e7" exitCode=0 Sep 08 18:54:18.083216 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:18.083165 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" event={"ID":"40ad8f7d-0594-4492-841f-966d8a3b9994","Type":"ContainerDied","Data":"04dc394c5eceaa5b5970d471233656c14ba577ac7e8a298d8501b8f8d0a360e7"} Sep 08 18:54:18.083702 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:18.083682 2553 scope.go:117] "RemoveContainer" containerID="04dc394c5eceaa5b5970d471233656c14ba577ac7e8a298d8501b8f8d0a360e7" Sep 08 18:54:18.084847 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:18.084820 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw" event={"ID":"0f55817b-93a2-43a3-807c-add0c90ff8d0","Type":"ContainerStarted","Data":"d5f75c14430a9e759914adc837c3d54e7ac562d78c167c3b0c00afa523a5548f"} Sep 08 18:54:18.085020 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:18.084889 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:54:18.085020 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:18.084981 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:54:18.085155 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:18.085030 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:54:19.092443 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:19.092406 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" event={"ID":"40ad8f7d-0594-4492-841f-966d8a3b9994","Type":"ContainerStarted","Data":"75d01308d6997a967a7c1318cd20fa0193101872decd77f25885142361932a90"} Sep 08 18:54:19.096634 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:19.096577 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw" event={"ID":"0f55817b-93a2-43a3-807c-add0c90ff8d0","Type":"ContainerStarted","Data":"230b0b1976f05308948afd6cea5c89303a232f7542245c24ba28826eabb60d9e"} Sep 08 18:54:19.345972 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:19.345885 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:54:19.347013 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:19.346986 2553 patch_prober.go:28] interesting pod/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Sep 08 18:54:19.347206 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:19.347181 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" podUID="40ad8f7d-0594-4492-841f-966d8a3b9994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Sep 08 18:54:20.103377 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:20.103331 2553 generic.go:356] "Generic (PLEG): container finished" podID="0f55817b-93a2-43a3-807c-add0c90ff8d0" containerID="230b0b1976f05308948afd6cea5c89303a232f7542245c24ba28826eabb60d9e" exitCode=0 Sep 08 18:54:20.103757 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:20.103416 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw" event={"ID":"0f55817b-93a2-43a3-807c-add0c90ff8d0","Type":"ContainerDied","Data":"230b0b1976f05308948afd6cea5c89303a232f7542245c24ba28826eabb60d9e"} Sep 08 18:54:20.345718 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:20.345674 2553 patch_prober.go:28] interesting pod/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Sep 08 18:54:20.345911 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:20.345743 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" podUID="40ad8f7d-0594-4492-841f-966d8a3b9994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Sep 08 18:54:21.286929 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:21.286904 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw" Sep 08 18:54:21.347261 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:21.346781 2553 patch_prober.go:28] interesting pod/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Sep 08 18:54:21.347261 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:21.346863 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" podUID="40ad8f7d-0594-4492-841f-966d8a3b9994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Sep 08 18:54:21.369309 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:21.369224 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f55817b-93a2-43a3-807c-add0c90ff8d0-maas-api-service-ca\") pod \"0f55817b-93a2-43a3-807c-add0c90ff8d0\" (UID: \"0f55817b-93a2-43a3-807c-add0c90ff8d0\") " Sep 08 18:54:21.369574 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:21.369550 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f55817b-93a2-43a3-807c-add0c90ff8d0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f55817b-93a2-43a3-807c-add0c90ff8d0" (UID: "0f55817b-93a2-43a3-807c-add0c90ff8d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:54:21.470286 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:21.470257 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f55817b-93a2-43a3-807c-add0c90ff8d0-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:22.116693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:22.116659 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw" Sep 08 18:54:22.116859 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:22.116696 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw" event={"ID":"0f55817b-93a2-43a3-807c-add0c90ff8d0","Type":"ContainerDied","Data":"d5f75c14430a9e759914adc837c3d54e7ac562d78c167c3b0c00afa523a5548f"} Sep 08 18:54:22.116859 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:22.116745 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5f75c14430a9e759914adc837c3d54e7ac562d78c167c3b0c00afa523a5548f" Sep 08 18:54:22.345805 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:22.345761 2553 patch_prober.go:28] interesting pod/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Sep 08 18:54:22.346270 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:22.345858 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" podUID="40ad8f7d-0594-4492-841f-966d8a3b9994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Sep 08 18:54:23.283850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.283819 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6956efcb-vffkg"] Sep 08 18:54:23.284321 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.284301 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerName="istio-proxy" Sep 08 18:54:23.284321 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.284322 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerName="istio-proxy" Sep 08 18:54:23.284514 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.284350 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f55817b-93a2-43a3-807c-add0c90ff8d0" containerName="revoke-keys" Sep 08 18:54:23.284514 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.284359 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f55817b-93a2-43a3-807c-add0c90ff8d0" containerName="revoke-keys" Sep 08 18:54:23.284514 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.284479 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f55817b-93a2-43a3-807c-add0c90ff8d0" containerName="revoke-keys" Sep 08 18:54:23.284514 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.284494 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d224cc-495b-47ca-a003-51cfc8fc2722" containerName="istio-proxy" Sep 08 18:54:23.286760 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.286739 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6956efcb-vffkg" Sep 08 18:54:23.289984 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.289963 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 18:54:23.290093 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.289967 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 18:54:23.302601 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.302569 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6956efcb-vffkg"] Sep 08 18:54:23.350225 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.349804 2553 patch_prober.go:28] interesting pod/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Sep 08 18:54:23.350225 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.349862 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" podUID="40ad8f7d-0594-4492-841f-966d8a3b9994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Sep 08 18:54:23.388914 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.388884 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ef15c4-8d42-484a-ac22-d28dea528306-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6956efcb-vffkg\" (UID: \"32ef15c4-8d42-484a-ac22-d28dea528306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6956efcb-vffkg" Sep 08 18:54:23.489753 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.489718 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ef15c4-8d42-484a-ac22-d28dea528306-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6956efcb-vffkg\" (UID: \"32ef15c4-8d42-484a-ac22-d28dea528306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6956efcb-vffkg" Sep 08 18:54:23.490423 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.490403 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ef15c4-8d42-484a-ac22-d28dea528306-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-6956efcb-vffkg\" (UID: \"32ef15c4-8d42-484a-ac22-d28dea528306\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6956efcb-vffkg" Sep 08 18:54:23.596954 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.596874 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6956efcb-vffkg" Sep 08 18:54:23.899764 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:23.899740 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6956efcb-vffkg"] Sep 08 18:54:23.901210 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:54:23.901183 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ef15c4_8d42_484a_ac22_d28dea528306.slice/crio-85a06350acdc1a615680fc6263b96f96d5a9e9dd0000908b28dfd0f210857a26 WatchSource:0}: Error finding container 85a06350acdc1a615680fc6263b96f96d5a9e9dd0000908b28dfd0f210857a26: Status 404 returned error can't find the container with id 85a06350acdc1a615680fc6263b96f96d5a9e9dd0000908b28dfd0f210857a26 Sep 08 18:54:24.131355 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:24.131308 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6956efcb-vffkg" event={"ID":"32ef15c4-8d42-484a-ac22-d28dea528306","Type":"ContainerStarted","Data":"85a06350acdc1a615680fc6263b96f96d5a9e9dd0000908b28dfd0f210857a26"} Sep 08 18:54:24.441104 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:24.441070 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:54:24.441602 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:24.441474 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:54:24.442687 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:24.442658 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:54:24.877093 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:24.877065 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 18:54:24.878623 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:24.878592 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 18:54:24.881096 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:24.881080 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 18:54:24.882248 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:24.882226 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 18:54:25.138679 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:25.138651 2553 generic.go:356] "Generic (PLEG): container finished" podID="32ef15c4-8d42-484a-ac22-d28dea528306" containerID="7bae01e7a5b92e602d46630c610a1ff8cce512d2dd225b2f5608a73a8d1a4d46" exitCode=0 Sep 08 18:54:25.138851 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:25.138750 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6956efcb-vffkg" event={"ID":"32ef15c4-8d42-484a-ac22-d28dea528306","Type":"ContainerDied","Data":"7bae01e7a5b92e602d46630c610a1ff8cce512d2dd225b2f5608a73a8d1a4d46"} Sep 08 18:54:26.278037 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:26.278015 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6956efcb-vffkg" Sep 08 18:54:26.320182 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:26.320139 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ef15c4-8d42-484a-ac22-d28dea528306-maas-api-service-ca\") pod \"32ef15c4-8d42-484a-ac22-d28dea528306\" (UID: \"32ef15c4-8d42-484a-ac22-d28dea528306\") " Sep 08 18:54:26.320469 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:26.320444 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ef15c4-8d42-484a-ac22-d28dea528306-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "32ef15c4-8d42-484a-ac22-d28dea528306" (UID: "32ef15c4-8d42-484a-ac22-d28dea528306"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:54:26.421805 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:26.421731 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ef15c4-8d42-484a-ac22-d28dea528306-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:27.151524 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:27.151496 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6956efcb-vffkg" Sep 08 18:54:27.151682 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:27.151532 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6956efcb-vffkg" event={"ID":"32ef15c4-8d42-484a-ac22-d28dea528306","Type":"ContainerDied","Data":"85a06350acdc1a615680fc6263b96f96d5a9e9dd0000908b28dfd0f210857a26"} Sep 08 18:54:27.151682 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:27.151583 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85a06350acdc1a615680fc6263b96f96d5a9e9dd0000908b28dfd0f210857a26" Sep 08 18:54:28.306691 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:28.306654 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh"] Sep 08 18:54:28.307147 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:28.306954 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" podUID="876fc381-1574-468c-803b-5e6aa3bc311d" containerName="payload-pre-processing" containerID="cri-o://92a02ddc61d0f8fbbce63f9556837770a24e024dbbc0a522004a793a8008f71f" gracePeriod=30 Sep 08 18:54:28.501145 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:28.501107 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw"] Sep 08 18:54:28.505424 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:28.505396 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-235f65-zp5bw"] Sep 08 18:54:28.567111 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:28.567046 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" Sep 08 18:54:28.651131 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:28.651094 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876fc381-1574-468c-803b-5e6aa3bc311d-plugins-config-volume\") pod \"876fc381-1574-468c-803b-5e6aa3bc311d\" (UID: \"876fc381-1574-468c-803b-5e6aa3bc311d\") " Sep 08 18:54:28.651405 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:28.651208 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74ksj\" (UniqueName: \"kubernetes.io/projected/876fc381-1574-468c-803b-5e6aa3bc311d-kube-api-access-74ksj\") pod \"876fc381-1574-468c-803b-5e6aa3bc311d\" (UID: \"876fc381-1574-468c-803b-5e6aa3bc311d\") " Sep 08 18:54:28.652038 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:28.651609 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/876fc381-1574-468c-803b-5e6aa3bc311d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "876fc381-1574-468c-803b-5e6aa3bc311d" (UID: "876fc381-1574-468c-803b-5e6aa3bc311d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:54:28.653986 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:28.653956 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/876fc381-1574-468c-803b-5e6aa3bc311d-kube-api-access-74ksj" (OuterVolumeSpecName: "kube-api-access-74ksj") pod "876fc381-1574-468c-803b-5e6aa3bc311d" (UID: "876fc381-1574-468c-803b-5e6aa3bc311d"). InnerVolumeSpecName "kube-api-access-74ksj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:54:28.744987 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:28.744957 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f55817b-93a2-43a3-807c-add0c90ff8d0" path="/var/lib/kubelet/pods/0f55817b-93a2-43a3-807c-add0c90ff8d0/volumes" Sep 08 18:54:28.752442 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:28.752258 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74ksj\" (UniqueName: \"kubernetes.io/projected/876fc381-1574-468c-803b-5e6aa3bc311d-kube-api-access-74ksj\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:28.752442 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:28.752283 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/876fc381-1574-468c-803b-5e6aa3bc311d-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:29.165507 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:29.165475 2553 generic.go:356] "Generic (PLEG): container finished" podID="876fc381-1574-468c-803b-5e6aa3bc311d" containerID="92a02ddc61d0f8fbbce63f9556837770a24e024dbbc0a522004a793a8008f71f" exitCode=0 Sep 08 18:54:29.165673 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:29.165514 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" Sep 08 18:54:29.165673 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:29.165521 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" event={"ID":"876fc381-1574-468c-803b-5e6aa3bc311d","Type":"ContainerDied","Data":"92a02ddc61d0f8fbbce63f9556837770a24e024dbbc0a522004a793a8008f71f"} Sep 08 18:54:29.165673 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:29.165568 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh" event={"ID":"876fc381-1574-468c-803b-5e6aa3bc311d","Type":"ContainerDied","Data":"76fc803d2b6e6ecd0bccc88f204d663c8cba39969937de649497a8bb2c4f7739"} Sep 08 18:54:29.165673 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:29.165590 2553 scope.go:117] "RemoveContainer" containerID="92a02ddc61d0f8fbbce63f9556837770a24e024dbbc0a522004a793a8008f71f" Sep 08 18:54:29.175108 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:29.175089 2553 scope.go:117] "RemoveContainer" containerID="92a02ddc61d0f8fbbce63f9556837770a24e024dbbc0a522004a793a8008f71f" Sep 08 18:54:29.175377 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:54:29.175341 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a02ddc61d0f8fbbce63f9556837770a24e024dbbc0a522004a793a8008f71f\": container with ID starting with 92a02ddc61d0f8fbbce63f9556837770a24e024dbbc0a522004a793a8008f71f not found: ID does not exist" containerID="92a02ddc61d0f8fbbce63f9556837770a24e024dbbc0a522004a793a8008f71f" Sep 08 18:54:29.175486 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:29.175384 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a02ddc61d0f8fbbce63f9556837770a24e024dbbc0a522004a793a8008f71f"} err="failed to get container status \"92a02ddc61d0f8fbbce63f9556837770a24e024dbbc0a522004a793a8008f71f\": rpc error: code = NotFound desc = could not find container \"92a02ddc61d0f8fbbce63f9556837770a24e024dbbc0a522004a793a8008f71f\": container with ID starting with 92a02ddc61d0f8fbbce63f9556837770a24e024dbbc0a522004a793a8008f71f not found: ID does not exist" Sep 08 18:54:29.202012 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:29.201985 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh"] Sep 08 18:54:29.207219 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:29.207195 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-235f65-598b4d8c86-2pckh"] Sep 08 18:54:30.745050 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:30.745018 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="876fc381-1574-468c-803b-5e6aa3bc311d" path="/var/lib/kubelet/pods/876fc381-1574-468c-803b-5e6aa3bc311d/volumes" Sep 08 18:54:30.921881 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:30.921848 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6956efcb-vffkg"] Sep 08 18:54:30.924569 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:30.924548 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6956efcb-vffkg"] Sep 08 18:54:31.089184 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:31.089097 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm"] Sep 08 18:54:31.089346 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:31.089294 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" podUID="40ad8f7d-0594-4492-841f-966d8a3b9994" containerName="istio-proxy" containerID="cri-o://75d01308d6997a967a7c1318cd20fa0193101872decd77f25885142361932a90" gracePeriod=30 Sep 08 18:54:32.749725 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:32.749688 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ef15c4-8d42-484a-ac22-d28dea528306" path="/var/lib/kubelet/pods/32ef15c4-8d42-484a-ac22-d28dea528306/volumes" Sep 08 18:54:36.210722 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.210676 2553 generic.go:356] "Generic (PLEG): container finished" podID="40ad8f7d-0594-4492-841f-966d8a3b9994" containerID="75d01308d6997a967a7c1318cd20fa0193101872decd77f25885142361932a90" exitCode=0 Sep 08 18:54:36.210722 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.210708 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" event={"ID":"40ad8f7d-0594-4492-841f-966d8a3b9994","Type":"ContainerDied","Data":"75d01308d6997a967a7c1318cd20fa0193101872decd77f25885142361932a90"} Sep 08 18:54:36.211147 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.210780 2553 scope.go:117] "RemoveContainer" containerID="04dc394c5eceaa5b5970d471233656c14ba577ac7e8a298d8501b8f8d0a360e7" Sep 08 18:54:36.350627 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.350576 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:54:36.425602 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.425572 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-workload-socket\") pod \"40ad8f7d-0594-4492-841f-966d8a3b9994\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " Sep 08 18:54:36.425602 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.425606 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-credential-socket\") pod \"40ad8f7d-0594-4492-841f-966d8a3b9994\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " Sep 08 18:54:36.425826 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.425642 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-token\") pod \"40ad8f7d-0594-4492-841f-966d8a3b9994\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " Sep 08 18:54:36.425826 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.425704 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-data\") pod \"40ad8f7d-0594-4492-841f-966d8a3b9994\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " Sep 08 18:54:36.425826 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.425746 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-podinfo\") pod \"40ad8f7d-0594-4492-841f-966d8a3b9994\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " Sep 08 18:54:36.425826 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.425783 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsd4c\" (UniqueName: \"kubernetes.io/projected/40ad8f7d-0594-4492-841f-966d8a3b9994-kube-api-access-bsd4c\") pod \"40ad8f7d-0594-4492-841f-966d8a3b9994\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " Sep 08 18:54:36.425826 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.425818 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-envoy\") pod \"40ad8f7d-0594-4492-841f-966d8a3b9994\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " Sep 08 18:54:36.426091 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.425852 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ad8f7d-0594-4492-841f-966d8a3b9994-istiod-ca-cert\") pod \"40ad8f7d-0594-4492-841f-966d8a3b9994\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " Sep 08 18:54:36.426091 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.425895 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-workload-certs\") pod \"40ad8f7d-0594-4492-841f-966d8a3b9994\" (UID: \"40ad8f7d-0594-4492-841f-966d8a3b9994\") " Sep 08 18:54:36.426202 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.425888 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "40ad8f7d-0594-4492-841f-966d8a3b9994" (UID: "40ad8f7d-0594-4492-841f-966d8a3b9994"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:54:36.426202 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.425907 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "40ad8f7d-0594-4492-841f-966d8a3b9994" (UID: "40ad8f7d-0594-4492-841f-966d8a3b9994"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:54:36.426202 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.426018 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-data" (OuterVolumeSpecName: "istio-data") pod "40ad8f7d-0594-4492-841f-966d8a3b9994" (UID: "40ad8f7d-0594-4492-841f-966d8a3b9994"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:54:36.426354 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.426230 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:36.426354 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.426209 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40ad8f7d-0594-4492-841f-966d8a3b9994-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "40ad8f7d-0594-4492-841f-966d8a3b9994" (UID: "40ad8f7d-0594-4492-841f-966d8a3b9994"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:54:36.426354 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.426249 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:36.426354 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.426262 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:36.426354 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.426333 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "40ad8f7d-0594-4492-841f-966d8a3b9994" (UID: "40ad8f7d-0594-4492-841f-966d8a3b9994"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:54:36.428016 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.427985 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ad8f7d-0594-4492-841f-966d8a3b9994-kube-api-access-bsd4c" (OuterVolumeSpecName: "kube-api-access-bsd4c") pod "40ad8f7d-0594-4492-841f-966d8a3b9994" (UID: "40ad8f7d-0594-4492-841f-966d8a3b9994"). InnerVolumeSpecName "kube-api-access-bsd4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:54:36.428016 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.428004 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-token" (OuterVolumeSpecName: "istio-token") pod "40ad8f7d-0594-4492-841f-966d8a3b9994" (UID: "40ad8f7d-0594-4492-841f-966d8a3b9994"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:54:36.428249 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.428224 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "40ad8f7d-0594-4492-841f-966d8a3b9994" (UID: "40ad8f7d-0594-4492-841f-966d8a3b9994"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:54:36.428312 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.428254 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "40ad8f7d-0594-4492-841f-966d8a3b9994" (UID: "40ad8f7d-0594-4492-841f-966d8a3b9994"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 18:54:36.527420 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.527389 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:36.527420 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.527427 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:36.527420 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.527436 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:36.527750 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.527445 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsd4c\" (UniqueName: \"kubernetes.io/projected/40ad8f7d-0594-4492-841f-966d8a3b9994-kube-api-access-bsd4c\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:36.527750 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.527458 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ad8f7d-0594-4492-841f-966d8a3b9994-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:36.527750 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.527470 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ad8f7d-0594-4492-841f-966d8a3b9994-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:54:36.552285 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.552255 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s"] Sep 08 18:54:36.552683 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.552668 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40ad8f7d-0594-4492-841f-966d8a3b9994" containerName="istio-proxy" Sep 08 18:54:36.552727 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.552685 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="40ad8f7d-0594-4492-841f-966d8a3b9994" containerName="istio-proxy" Sep 08 18:54:36.552727 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.552702 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40ad8f7d-0594-4492-841f-966d8a3b9994" containerName="istio-proxy" Sep 08 18:54:36.552727 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.552709 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="40ad8f7d-0594-4492-841f-966d8a3b9994" containerName="istio-proxy" Sep 08 18:54:36.552727 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.552724 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ef15c4-8d42-484a-ac22-d28dea528306" containerName="revoke-keys" Sep 08 18:54:36.552875 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.552731 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ef15c4-8d42-484a-ac22-d28dea528306" containerName="revoke-keys" Sep 08 18:54:36.552875 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.552743 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="876fc381-1574-468c-803b-5e6aa3bc311d" containerName="payload-pre-processing" Sep 08 18:54:36.552875 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.552749 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="876fc381-1574-468c-803b-5e6aa3bc311d" containerName="payload-pre-processing" Sep 08 18:54:36.552875 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.552810 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="40ad8f7d-0594-4492-841f-966d8a3b9994" containerName="istio-proxy" Sep 08 18:54:36.552875 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.552818 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="40ad8f7d-0594-4492-841f-966d8a3b9994" containerName="istio-proxy" Sep 08 18:54:36.552875 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.552826 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ef15c4-8d42-484a-ac22-d28dea528306" containerName="revoke-keys" Sep 08 18:54:36.552875 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.552833 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="876fc381-1574-468c-803b-5e6aa3bc311d" containerName="payload-pre-processing" Sep 08 18:54:36.555028 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.555012 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.558300 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.558274 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-72dcc3-openshift-default-dockercfg-989w4\"" Sep 08 18:54:36.569707 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.569682 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s"] Sep 08 18:54:36.628032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.627999 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-podinfo\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.628225 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.628071 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gschh\" (UniqueName: \"kubernetes.io/projected/0f1f962f-5c18-4529-96d8-a8cfcb970e95-kube-api-access-gschh\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.628225 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.628100 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-credential-socket\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.628225 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.628172 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-token\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.628333 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.628256 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-data\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.628333 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.628315 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-workload-certs\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.628432 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.628342 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-workload-socket\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.628432 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.628395 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istiod-ca-cert\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.628432 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.628424 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-envoy\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.729407 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.729354 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-data\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.729589 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.729450 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-workload-certs\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.729589 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.729486 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-workload-socket\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.729589 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.729521 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istiod-ca-cert\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.729589 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.729551 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-envoy\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.729589 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.729580 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-podinfo\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.729855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.729641 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gschh\" (UniqueName: \"kubernetes.io/projected/0f1f962f-5c18-4529-96d8-a8cfcb970e95-kube-api-access-gschh\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.729855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.729667 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-credential-socket\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.729855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.729694 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-token\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.729855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.729792 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-data\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.729855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.729828 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-workload-certs\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.730077 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.729854 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-workload-socket\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.730131 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.730084 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-credential-socket\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.730235 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.730212 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istiod-ca-cert\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.731762 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.731736 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-envoy\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.732083 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.732066 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-podinfo\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.740025 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.740001 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-token\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.740114 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.740004 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gschh\" (UniqueName: \"kubernetes.io/projected/0f1f962f-5c18-4529-96d8-a8cfcb970e95-kube-api-access-gschh\") pod \"e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:36.866442 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:36.866412 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:37.146281 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:37.146257 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s"] Sep 08 18:54:37.147803 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:54:37.147774 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1f962f_5c18_4529_96d8_a8cfcb970e95.slice/crio-758170effb2b72bfd94760949d77213dd427c54cbc8e158c453fa4ee59ffad89 WatchSource:0}: Error finding container 758170effb2b72bfd94760949d77213dd427c54cbc8e158c453fa4ee59ffad89: Status 404 returned error can't find the container with id 758170effb2b72bfd94760949d77213dd427c54cbc8e158c453fa4ee59ffad89 Sep 08 18:54:37.150166 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:37.150123 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:54:37.150235 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:37.150211 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:54:37.150277 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:37.150252 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:54:37.219137 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:37.219112 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" Sep 08 18:54:37.219539 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:37.219138 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm" event={"ID":"40ad8f7d-0594-4492-841f-966d8a3b9994","Type":"ContainerDied","Data":"75c07183d016abe368d7e9248a2280237007a3f25c42215907cb89e1a3834c88"} Sep 08 18:54:37.219539 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:37.219243 2553 scope.go:117] "RemoveContainer" containerID="75d01308d6997a967a7c1318cd20fa0193101872decd77f25885142361932a90" Sep 08 18:54:37.221825 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:37.221795 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" event={"ID":"0f1f962f-5c18-4529-96d8-a8cfcb970e95","Type":"ContainerStarted","Data":"d9871c18770588bb942dca6c571366809a957b5d4eff8548e5b654f99a9cef3e"} Sep 08 18:54:37.221948 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:37.221852 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" event={"ID":"0f1f962f-5c18-4529-96d8-a8cfcb970e95","Type":"ContainerStarted","Data":"758170effb2b72bfd94760949d77213dd427c54cbc8e158c453fa4ee59ffad89"} Sep 08 18:54:37.256505 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:37.256472 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm"] Sep 08 18:54:37.259479 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:37.259452 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-6956efcb-openshift-default-647dcfc5b4-qbqbm"] Sep 08 18:54:37.286334 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:37.286268 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" podStartSLOduration=1.2862482210000001 podStartE2EDuration="1.286248221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:54:37.284121624 +0000 UTC m=+1212.990009857" watchObservedRunningTime="2026-09-08 18:54:37.286248221 +0000 UTC m=+1212.992136454" Sep 08 18:54:37.867012 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:37.866981 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:38.745380 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:38.745340 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40ad8f7d-0594-4492-841f-966d8a3b9994" path="/var/lib/kubelet/pods/40ad8f7d-0594-4492-841f-966d8a3b9994/volumes" Sep 08 18:54:38.871983 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:38.871957 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:39.235221 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:39.235187 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:54:39.236122 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:54:39.236101 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:55:00.355940 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:00.355908 2553 generic.go:356] "Generic (PLEG): container finished" podID="0f1f962f-5c18-4529-96d8-a8cfcb970e95" containerID="d9871c18770588bb942dca6c571366809a957b5d4eff8548e5b654f99a9cef3e" exitCode=0 Sep 08 18:55:00.356387 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:00.355992 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" event={"ID":"0f1f962f-5c18-4529-96d8-a8cfcb970e95","Type":"ContainerDied","Data":"d9871c18770588bb942dca6c571366809a957b5d4eff8548e5b654f99a9cef3e"} Sep 08 18:55:00.356451 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:00.356408 2553 scope.go:117] "RemoveContainer" containerID="d9871c18770588bb942dca6c571366809a957b5d4eff8548e5b654f99a9cef3e" Sep 08 18:55:00.357483 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:00.357455 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:55:00.357561 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:00.357523 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:55:00.357561 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:00.357556 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:55:01.364163 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:01.364127 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" event={"ID":"0f1f962f-5c18-4529-96d8-a8cfcb970e95","Type":"ContainerStarted","Data":"8f7286b0e3970fa3ea28144f7c20da7c4cca1de49be087a3c6feaa16d6449336"} Sep 08 18:55:01.867167 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:01.867126 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:55:01.867356 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:01.867334 2553 patch_prober.go:28] interesting pod/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 08 18:55:01.867435 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:01.867414 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" podUID="0f1f962f-5c18-4529-96d8-a8cfcb970e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 08 18:55:02.867387 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:02.867333 2553 patch_prober.go:28] interesting pod/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 08 18:55:02.867771 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:02.867434 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" podUID="0f1f962f-5c18-4529-96d8-a8cfcb970e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 08 18:55:03.867783 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:03.867741 2553 patch_prober.go:28] interesting pod/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 08 18:55:03.868175 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:03.867812 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" podUID="0f1f962f-5c18-4529-96d8-a8cfcb970e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 08 18:55:04.867286 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:04.867245 2553 patch_prober.go:28] interesting pod/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 08 18:55:04.867492 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:04.867306 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" podUID="0f1f962f-5c18-4529-96d8-a8cfcb970e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 08 18:55:05.867076 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:05.867037 2553 patch_prober.go:28] interesting pod/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 08 18:55:05.867482 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:05.867091 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" podUID="0f1f962f-5c18-4529-96d8-a8cfcb970e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 08 18:55:06.870975 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:06.870942 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:55:06.871438 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:06.871227 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:55:06.872011 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:06.871990 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:55:23.239655 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:23.239569 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt"] Sep 08 18:55:23.242212 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:23.242188 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt" Sep 08 18:55:23.245125 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:23.245103 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 18:55:23.245125 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:23.245116 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 18:55:23.255708 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:23.255675 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt"] Sep 08 18:55:23.349877 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:23.349845 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/264e238a-e277-416f-b775-6c5b184251c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt\" (UID: \"264e238a-e277-416f-b775-6c5b184251c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt" Sep 08 18:55:23.450765 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:23.450728 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/264e238a-e277-416f-b775-6c5b184251c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt\" (UID: \"264e238a-e277-416f-b775-6c5b184251c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt" Sep 08 18:55:23.451391 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:23.451341 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/264e238a-e277-416f-b775-6c5b184251c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt\" (UID: \"264e238a-e277-416f-b775-6c5b184251c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt" Sep 08 18:55:23.553340 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:23.553257 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt" Sep 08 18:55:23.827463 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:23.827427 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt"] Sep 08 18:55:23.828664 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:55:23.828641 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264e238a_e277_416f_b775_6c5b184251c1.slice/crio-aee699e0a61e4d5959a14ef14705cfef63c4e1319b9298c90fa5f8c097eeaed9 WatchSource:0}: Error finding container aee699e0a61e4d5959a14ef14705cfef63c4e1319b9298c90fa5f8c097eeaed9: Status 404 returned error can't find the container with id aee699e0a61e4d5959a14ef14705cfef63c4e1319b9298c90fa5f8c097eeaed9 Sep 08 18:55:24.496519 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:24.496483 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt" event={"ID":"264e238a-e277-416f-b775-6c5b184251c1","Type":"ContainerStarted","Data":"83fcbdeceb13c145674fdfa4fe61b3107f2e7d2a1d60ed26981decc3f5822b87"} Sep 08 18:55:24.496947 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:24.496543 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt" event={"ID":"264e238a-e277-416f-b775-6c5b184251c1","Type":"ContainerStarted","Data":"aee699e0a61e4d5959a14ef14705cfef63c4e1319b9298c90fa5f8c097eeaed9"} Sep 08 18:55:24.513557 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:24.513499 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt" podStartSLOduration=1.513481515 podStartE2EDuration="1.513481515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:55:24.513307191 +0000 UTC m=+1260.219195414" watchObservedRunningTime="2026-09-08 18:55:24.513481515 +0000 UTC m=+1260.219369748" Sep 08 18:55:25.503283 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:25.503243 2553 generic.go:356] "Generic (PLEG): container finished" podID="264e238a-e277-416f-b775-6c5b184251c1" containerID="83fcbdeceb13c145674fdfa4fe61b3107f2e7d2a1d60ed26981decc3f5822b87" exitCode=0 Sep 08 18:55:25.503682 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:25.503292 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt" event={"ID":"264e238a-e277-416f-b775-6c5b184251c1","Type":"ContainerDied","Data":"83fcbdeceb13c145674fdfa4fe61b3107f2e7d2a1d60ed26981decc3f5822b87"} Sep 08 18:55:26.642829 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:26.642807 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt" Sep 08 18:55:26.780129 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:26.780040 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/264e238a-e277-416f-b775-6c5b184251c1-maas-api-service-ca\") pod \"264e238a-e277-416f-b775-6c5b184251c1\" (UID: \"264e238a-e277-416f-b775-6c5b184251c1\") " Sep 08 18:55:26.780411 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:26.780357 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264e238a-e277-416f-b775-6c5b184251c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "264e238a-e277-416f-b775-6c5b184251c1" (UID: "264e238a-e277-416f-b775-6c5b184251c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:55:26.881247 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:26.881209 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/264e238a-e277-416f-b775-6c5b184251c1-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:55:27.515342 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:27.515311 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt" Sep 08 18:55:27.515602 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:27.515355 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt" event={"ID":"264e238a-e277-416f-b775-6c5b184251c1","Type":"ContainerDied","Data":"aee699e0a61e4d5959a14ef14705cfef63c4e1319b9298c90fa5f8c097eeaed9"} Sep 08 18:55:27.515602 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:27.515426 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aee699e0a61e4d5959a14ef14705cfef63c4e1319b9298c90fa5f8c097eeaed9" Sep 08 18:55:33.849993 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:33.849952 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt"] Sep 08 18:55:33.852216 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:33.852184 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-72dcc3-5fxvt"] Sep 08 18:55:34.750103 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:34.750071 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="264e238a-e277-416f-b775-6c5b184251c1" path="/var/lib/kubelet/pods/264e238a-e277-416f-b775-6c5b184251c1/volumes" Sep 08 18:55:40.437919 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:40.437884 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s"] Sep 08 18:55:40.438320 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:40.438101 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" podUID="0f1f962f-5c18-4529-96d8-a8cfcb970e95" containerName="istio-proxy" containerID="cri-o://8f7286b0e3970fa3ea28144f7c20da7c4cca1de49be087a3c6feaa16d6449336" gracePeriod=30 Sep 08 18:55:45.627838 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.627801 2553 generic.go:356] "Generic (PLEG): container finished" podID="0f1f962f-5c18-4529-96d8-a8cfcb970e95" containerID="8f7286b0e3970fa3ea28144f7c20da7c4cca1de49be087a3c6feaa16d6449336" exitCode=0 Sep 08 18:55:45.628219 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.627875 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" event={"ID":"0f1f962f-5c18-4529-96d8-a8cfcb970e95","Type":"ContainerDied","Data":"8f7286b0e3970fa3ea28144f7c20da7c4cca1de49be087a3c6feaa16d6449336"} Sep 08 18:55:45.628219 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.627967 2553 scope.go:117] "RemoveContainer" containerID="d9871c18770588bb942dca6c571366809a957b5d4eff8548e5b654f99a9cef3e" Sep 08 18:55:45.685182 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.685151 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:55:45.744290 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744216 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-workload-certs\") pod \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " Sep 08 18:55:45.744290 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744269 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-credential-socket\") pod \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " Sep 08 18:55:45.744290 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744288 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-workload-socket\") pod \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " Sep 08 18:55:45.744644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744322 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-data\") pod \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " Sep 08 18:55:45.744644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744337 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istiod-ca-cert\") pod \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " Sep 08 18:55:45.744644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744400 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-envoy\") pod \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " Sep 08 18:55:45.744644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744433 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-podinfo\") pod \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " Sep 08 18:55:45.744644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744481 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gschh\" (UniqueName: \"kubernetes.io/projected/0f1f962f-5c18-4529-96d8-a8cfcb970e95-kube-api-access-gschh\") pod \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " Sep 08 18:55:45.744644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744515 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-token\") pod \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\" (UID: \"0f1f962f-5c18-4529-96d8-a8cfcb970e95\") " Sep 08 18:55:45.744644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744595 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f1f962f-5c18-4529-96d8-a8cfcb970e95" (UID: "0f1f962f-5c18-4529-96d8-a8cfcb970e95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:55:45.744644 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744621 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f1f962f-5c18-4529-96d8-a8cfcb970e95" (UID: "0f1f962f-5c18-4529-96d8-a8cfcb970e95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:55:45.745042 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744666 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f1f962f-5c18-4529-96d8-a8cfcb970e95" (UID: "0f1f962f-5c18-4529-96d8-a8cfcb970e95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:55:45.745042 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744682 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f1f962f-5c18-4529-96d8-a8cfcb970e95" (UID: "0f1f962f-5c18-4529-96d8-a8cfcb970e95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:55:45.745042 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744720 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f1f962f-5c18-4529-96d8-a8cfcb970e95" (UID: "0f1f962f-5c18-4529-96d8-a8cfcb970e95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:55:45.745042 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744886 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:55:45.745042 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744906 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:55:45.745042 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744920 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:55:45.745042 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744932 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:55:45.745042 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.744940 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:55:45.746838 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.746810 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f1f962f-5c18-4529-96d8-a8cfcb970e95" (UID: "0f1f962f-5c18-4529-96d8-a8cfcb970e95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:55:45.746838 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.746829 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f1f962f-5c18-4529-96d8-a8cfcb970e95" (UID: "0f1f962f-5c18-4529-96d8-a8cfcb970e95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:55:45.746987 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.746857 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1f962f-5c18-4529-96d8-a8cfcb970e95-kube-api-access-gschh" (OuterVolumeSpecName: "kube-api-access-gschh") pod "0f1f962f-5c18-4529-96d8-a8cfcb970e95" (UID: "0f1f962f-5c18-4529-96d8-a8cfcb970e95"). InnerVolumeSpecName "kube-api-access-gschh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:55:45.746987 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.746882 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f1f962f-5c18-4529-96d8-a8cfcb970e95" (UID: "0f1f962f-5c18-4529-96d8-a8cfcb970e95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 18:55:45.845720 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.845694 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:55:45.845720 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.845720 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:55:45.845876 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.845729 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gschh\" (UniqueName: \"kubernetes.io/projected/0f1f962f-5c18-4529-96d8-a8cfcb970e95-kube-api-access-gschh\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:55:45.845876 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:45.845738 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f1f962f-5c18-4529-96d8-a8cfcb970e95-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:55:46.634865 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:46.634831 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" event={"ID":"0f1f962f-5c18-4529-96d8-a8cfcb970e95","Type":"ContainerDied","Data":"758170effb2b72bfd94760949d77213dd427c54cbc8e158c453fa4ee59ffad89"} Sep 08 18:55:46.634865 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:46.634853 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s" Sep 08 18:55:46.635352 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:46.634895 2553 scope.go:117] "RemoveContainer" containerID="8f7286b0e3970fa3ea28144f7c20da7c4cca1de49be087a3c6feaa16d6449336" Sep 08 18:55:46.659169 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:46.659136 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s"] Sep 08 18:55:46.664297 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:46.664275 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-72dcc3-openshift-default-746d98d88-btm7s"] Sep 08 18:55:46.750790 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:55:46.750758 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f1f962f-5c18-4529-96d8-a8cfcb970e95" path="/var/lib/kubelet/pods/0f1f962f-5c18-4529-96d8-a8cfcb970e95/volumes" Sep 08 18:56:08.625344 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.625308 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s"] Sep 08 18:56:08.625990 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.625871 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1f962f-5c18-4529-96d8-a8cfcb970e95" containerName="istio-proxy" Sep 08 18:56:08.625990 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.625888 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1f962f-5c18-4529-96d8-a8cfcb970e95" containerName="istio-proxy" Sep 08 18:56:08.625990 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.625899 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264e238a-e277-416f-b775-6c5b184251c1" containerName="revoke-keys" Sep 08 18:56:08.625990 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.625907 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="264e238a-e277-416f-b775-6c5b184251c1" containerName="revoke-keys" Sep 08 18:56:08.626228 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.626017 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1f962f-5c18-4529-96d8-a8cfcb970e95" containerName="istio-proxy" Sep 08 18:56:08.626228 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.626031 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="264e238a-e277-416f-b775-6c5b184251c1" containerName="revoke-keys" Sep 08 18:56:08.628389 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.628337 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.631139 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.631110 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-64effff4-openshift-default-dockercfg-85fpd\"" Sep 08 18:56:08.641194 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.641171 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s"] Sep 08 18:56:08.741719 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.741690 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-workload-certs\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.741914 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.741739 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgv54\" (UniqueName: \"kubernetes.io/projected/18d4a538-ab50-4491-a6ed-d8edea7dc906-kube-api-access-lgv54\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.741914 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.741768 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-workload-socket\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.741914 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.741791 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d4a538-ab50-4491-a6ed-d8edea7dc906-istiod-ca-cert\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.741914 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.741822 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-credential-socket\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.742125 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.741938 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-token\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.742125 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.742039 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-envoy\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.742125 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.742107 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-data\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.742255 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.742152 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-podinfo\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.843562 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.843531 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-workload-certs\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.843747 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.843586 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgv54\" (UniqueName: \"kubernetes.io/projected/18d4a538-ab50-4491-a6ed-d8edea7dc906-kube-api-access-lgv54\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.843747 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.843616 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-workload-socket\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.843747 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.843652 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d4a538-ab50-4491-a6ed-d8edea7dc906-istiod-ca-cert\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.843925 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.843837 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-credential-socket\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.843925 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.843872 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-token\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.843925 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.843916 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-envoy\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.844080 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.843973 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-data\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.844080 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.844006 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-podinfo\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.844080 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.844019 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-workload-certs\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.844232 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.844169 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-workload-socket\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.844732 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.844459 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d4a538-ab50-4491-a6ed-d8edea7dc906-istiod-ca-cert\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.844851 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.844747 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-credential-socket\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.844947 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.844923 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-data\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.846518 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.846491 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-envoy\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.846914 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.846896 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-podinfo\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.854008 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.853983 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-token\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.854008 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.854000 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgv54\" (UniqueName: \"kubernetes.io/projected/18d4a538-ab50-4491-a6ed-d8edea7dc906-kube-api-access-lgv54\") pod \"e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:08.938695 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:08.938619 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:09.212735 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:09.212695 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s"] Sep 08 18:56:09.213552 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:56:09.213512 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d4a538_ab50_4491_a6ed_d8edea7dc906.slice/crio-d8933d4a11cbfafcff2d1e4fce708dbdfb7d49f76dde84ce619a4fbbdaa64d7f WatchSource:0}: Error finding container d8933d4a11cbfafcff2d1e4fce708dbdfb7d49f76dde84ce619a4fbbdaa64d7f: Status 404 returned error can't find the container with id d8933d4a11cbfafcff2d1e4fce708dbdfb7d49f76dde84ce619a4fbbdaa64d7f Sep 08 18:56:09.215835 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:09.215793 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:56:09.215917 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:09.215885 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:56:09.216011 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:09.215923 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:56:09.772843 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:09.772804 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" event={"ID":"18d4a538-ab50-4491-a6ed-d8edea7dc906","Type":"ContainerStarted","Data":"518a6d7895e12f3676b4c20630ff20d00036128a4cc5e65e45294e0f97eeff9e"} Sep 08 18:56:09.773220 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:09.772906 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" event={"ID":"18d4a538-ab50-4491-a6ed-d8edea7dc906","Type":"ContainerStarted","Data":"d8933d4a11cbfafcff2d1e4fce708dbdfb7d49f76dde84ce619a4fbbdaa64d7f"} Sep 08 18:56:09.797155 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:09.797101 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" podStartSLOduration=1.797085595 podStartE2EDuration="1.797085595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:56:09.79468127 +0000 UTC m=+1305.500569514" watchObservedRunningTime="2026-09-08 18:56:09.797085595 +0000 UTC m=+1305.502973828" Sep 08 18:56:09.939060 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:09.939027 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:10.944561 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:10.944530 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:11.784392 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:11.784341 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:11.785358 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:11.785337 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:19.697547 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:19.697506 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh"] Sep 08 18:56:19.698110 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:19.698091 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1f962f-5c18-4529-96d8-a8cfcb970e95" containerName="istio-proxy" Sep 08 18:56:19.698159 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:19.698114 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1f962f-5c18-4529-96d8-a8cfcb970e95" containerName="istio-proxy" Sep 08 18:56:19.698209 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:19.698197 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1f962f-5c18-4529-96d8-a8cfcb970e95" containerName="istio-proxy" Sep 08 18:56:19.700865 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:19.700844 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh" Sep 08 18:56:19.703645 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:19.703616 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 18:56:19.703887 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:19.703867 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 18:56:19.706036 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:19.706016 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh"] Sep 08 18:56:19.848056 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:19.848012 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48fa7b97-3831-462a-84af-8cc87f2d2de6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh\" (UID: \"48fa7b97-3831-462a-84af-8cc87f2d2de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh" Sep 08 18:56:19.949687 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:19.949582 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48fa7b97-3831-462a-84af-8cc87f2d2de6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh\" (UID: \"48fa7b97-3831-462a-84af-8cc87f2d2de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh" Sep 08 18:56:19.950447 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:19.950415 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48fa7b97-3831-462a-84af-8cc87f2d2de6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh\" (UID: \"48fa7b97-3831-462a-84af-8cc87f2d2de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh" Sep 08 18:56:20.014797 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:20.014761 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh" Sep 08 18:56:20.305543 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:20.305516 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh"] Sep 08 18:56:20.306842 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:56:20.306816 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fa7b97_3831_462a_84af_8cc87f2d2de6.slice/crio-178ebc2ce81a4f76ee831083bf5999414994322703662ef30f6419f56460eee3 WatchSource:0}: Error finding container 178ebc2ce81a4f76ee831083bf5999414994322703662ef30f6419f56460eee3: Status 404 returned error can't find the container with id 178ebc2ce81a4f76ee831083bf5999414994322703662ef30f6419f56460eee3 Sep 08 18:56:20.840146 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:20.840088 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh" event={"ID":"48fa7b97-3831-462a-84af-8cc87f2d2de6","Type":"ContainerStarted","Data":"149b6ef2e3717abf5e1ddbef93d9daa8d82171e39b1db100d2cba377a403107a"} Sep 08 18:56:20.840630 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:20.840171 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh" event={"ID":"48fa7b97-3831-462a-84af-8cc87f2d2de6","Type":"ContainerStarted","Data":"178ebc2ce81a4f76ee831083bf5999414994322703662ef30f6419f56460eee3"} Sep 08 18:56:20.867998 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:20.867946 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh" podStartSLOduration=1.86793143 podStartE2EDuration="1.86793143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:56:20.865788333 +0000 UTC m=+1316.571676571" watchObservedRunningTime="2026-09-08 18:56:20.86793143 +0000 UTC m=+1316.573819662" Sep 08 18:56:21.857599 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:21.857558 2553 generic.go:356] "Generic (PLEG): container finished" podID="48fa7b97-3831-462a-84af-8cc87f2d2de6" containerID="149b6ef2e3717abf5e1ddbef93d9daa8d82171e39b1db100d2cba377a403107a" exitCode=0 Sep 08 18:56:21.858094 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:21.857625 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh" event={"ID":"48fa7b97-3831-462a-84af-8cc87f2d2de6","Type":"ContainerDied","Data":"149b6ef2e3717abf5e1ddbef93d9daa8d82171e39b1db100d2cba377a403107a"} Sep 08 18:56:23.021007 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:23.020982 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh" Sep 08 18:56:23.079103 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:23.079073 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48fa7b97-3831-462a-84af-8cc87f2d2de6-maas-api-service-ca\") pod \"48fa7b97-3831-462a-84af-8cc87f2d2de6\" (UID: \"48fa7b97-3831-462a-84af-8cc87f2d2de6\") " Sep 08 18:56:23.079403 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:23.079379 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fa7b97-3831-462a-84af-8cc87f2d2de6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48fa7b97-3831-462a-84af-8cc87f2d2de6" (UID: "48fa7b97-3831-462a-84af-8cc87f2d2de6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:56:23.180555 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:23.180516 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48fa7b97-3831-462a-84af-8cc87f2d2de6-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:56:23.870721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:23.870692 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh" Sep 08 18:56:23.870901 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:23.870719 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh" event={"ID":"48fa7b97-3831-462a-84af-8cc87f2d2de6","Type":"ContainerDied","Data":"178ebc2ce81a4f76ee831083bf5999414994322703662ef30f6419f56460eee3"} Sep 08 18:56:23.870901 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:23.870762 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="178ebc2ce81a4f76ee831083bf5999414994322703662ef30f6419f56460eee3" Sep 08 18:56:27.825566 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:27.825510 2553 patch_prober.go:28] interesting pod/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 18:56:27.825935 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:27.825589 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 18:56:28.901619 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:28.901586 2553 generic.go:356] "Generic (PLEG): container finished" podID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerID="518a6d7895e12f3676b4c20630ff20d00036128a4cc5e65e45294e0f97eeff9e" exitCode=0 Sep 08 18:56:28.902036 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:28.901658 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" event={"ID":"18d4a538-ab50-4491-a6ed-d8edea7dc906","Type":"ContainerDied","Data":"518a6d7895e12f3676b4c20630ff20d00036128a4cc5e65e45294e0f97eeff9e"} Sep 08 18:56:28.902036 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:28.902034 2553 scope.go:117] "RemoveContainer" containerID="518a6d7895e12f3676b4c20630ff20d00036128a4cc5e65e45294e0f97eeff9e" Sep 08 18:56:28.903109 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:28.903073 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:56:28.903207 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:28.903147 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:56:28.903207 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:28.903180 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:56:29.910188 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:29.910156 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" event={"ID":"18d4a538-ab50-4491-a6ed-d8edea7dc906","Type":"ContainerStarted","Data":"cd5089be0fe6f7a91b84287d7f0b68ec79c2afac1284d7436e3dbafaf18ad711"} Sep 08 18:56:29.939699 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:29.939656 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:29.939892 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:29.939783 2553 patch_prober.go:28] interesting pod/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 18:56:29.939892 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:29.939825 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 18:56:30.290186 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.290101 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w"] Sep 08 18:56:30.290386 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.290342 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" podUID="4b2c3088-389c-4794-b7cd-aedcf837398e" containerName="payload-processing" containerID="cri-o://629d108319c2aebac5eb1451e50b12ac1d756a43f4fc53ea13a07a4cd39aeaf8" gracePeriod=30 Sep 08 18:56:30.298607 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.298575 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw"] Sep 08 18:56:30.298965 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.298789 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" podUID="99bd63ad-bd5b-4e0b-ae93-70b20ab0310f" containerName="payload-pre-processing" containerID="cri-o://5603c09c604cc0b098ae045748ee994de193a22680f8e20d63dd97956a5e37bb" gracePeriod=30 Sep 08 18:56:30.484211 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.484173 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh"] Sep 08 18:56:30.488272 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.487988 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-52ba8b-7zsnh"] Sep 08 18:56:30.635804 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.634537 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" Sep 08 18:56:30.652498 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.651524 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" Sep 08 18:56:30.756837 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.756803 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48fa7b97-3831-462a-84af-8cc87f2d2de6" path="/var/lib/kubelet/pods/48fa7b97-3831-462a-84af-8cc87f2d2de6/volumes" Sep 08 18:56:30.761565 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.761539 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2c3088-389c-4794-b7cd-aedcf837398e-plugins-config-volume\") pod \"4b2c3088-389c-4794-b7cd-aedcf837398e\" (UID: \"4b2c3088-389c-4794-b7cd-aedcf837398e\") " Sep 08 18:56:30.761736 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.761674 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99bd63ad-bd5b-4e0b-ae93-70b20ab0310f-plugins-config-volume\") pod \"99bd63ad-bd5b-4e0b-ae93-70b20ab0310f\" (UID: \"99bd63ad-bd5b-4e0b-ae93-70b20ab0310f\") " Sep 08 18:56:30.761907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.761883 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlvnd\" (UniqueName: \"kubernetes.io/projected/99bd63ad-bd5b-4e0b-ae93-70b20ab0310f-kube-api-access-nlvnd\") pod \"99bd63ad-bd5b-4e0b-ae93-70b20ab0310f\" (UID: \"99bd63ad-bd5b-4e0b-ae93-70b20ab0310f\") " Sep 08 18:56:30.761986 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.761946 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5rht\" (UniqueName: \"kubernetes.io/projected/4b2c3088-389c-4794-b7cd-aedcf837398e-kube-api-access-v5rht\") pod \"4b2c3088-389c-4794-b7cd-aedcf837398e\" (UID: \"4b2c3088-389c-4794-b7cd-aedcf837398e\") " Sep 08 18:56:30.762067 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.762046 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99bd63ad-bd5b-4e0b-ae93-70b20ab0310f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99bd63ad-bd5b-4e0b-ae93-70b20ab0310f" (UID: "99bd63ad-bd5b-4e0b-ae93-70b20ab0310f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:56:30.762116 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.762070 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2c3088-389c-4794-b7cd-aedcf837398e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b2c3088-389c-4794-b7cd-aedcf837398e" (UID: "4b2c3088-389c-4794-b7cd-aedcf837398e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:56:30.762286 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.762263 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2c3088-389c-4794-b7cd-aedcf837398e-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:56:30.762286 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.762285 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99bd63ad-bd5b-4e0b-ae93-70b20ab0310f-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:56:30.764009 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.763982 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2c3088-389c-4794-b7cd-aedcf837398e-kube-api-access-v5rht" (OuterVolumeSpecName: "kube-api-access-v5rht") pod "4b2c3088-389c-4794-b7cd-aedcf837398e" (UID: "4b2c3088-389c-4794-b7cd-aedcf837398e"). InnerVolumeSpecName "kube-api-access-v5rht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:56:30.764128 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.764085 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99bd63ad-bd5b-4e0b-ae93-70b20ab0310f-kube-api-access-nlvnd" (OuterVolumeSpecName: "kube-api-access-nlvnd") pod "99bd63ad-bd5b-4e0b-ae93-70b20ab0310f" (UID: "99bd63ad-bd5b-4e0b-ae93-70b20ab0310f"). InnerVolumeSpecName "kube-api-access-nlvnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:56:30.863033 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.862936 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5rht\" (UniqueName: \"kubernetes.io/projected/4b2c3088-389c-4794-b7cd-aedcf837398e-kube-api-access-v5rht\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:56:30.863033 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.862979 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlvnd\" (UniqueName: \"kubernetes.io/projected/99bd63ad-bd5b-4e0b-ae93-70b20ab0310f-kube-api-access-nlvnd\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:56:30.923653 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.923617 2553 generic.go:356] "Generic (PLEG): container finished" podID="4b2c3088-389c-4794-b7cd-aedcf837398e" containerID="629d108319c2aebac5eb1451e50b12ac1d756a43f4fc53ea13a07a4cd39aeaf8" exitCode=0 Sep 08 18:56:30.924115 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.923677 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" Sep 08 18:56:30.924115 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.923706 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" event={"ID":"4b2c3088-389c-4794-b7cd-aedcf837398e","Type":"ContainerDied","Data":"629d108319c2aebac5eb1451e50b12ac1d756a43f4fc53ea13a07a4cd39aeaf8"} Sep 08 18:56:30.924115 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.923800 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w" event={"ID":"4b2c3088-389c-4794-b7cd-aedcf837398e","Type":"ContainerDied","Data":"d7d6692603d9d97711c23e3abb461f4bf2076c077a1656fc4fe7861e14b7042a"} Sep 08 18:56:30.924115 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.923835 2553 scope.go:117] "RemoveContainer" containerID="629d108319c2aebac5eb1451e50b12ac1d756a43f4fc53ea13a07a4cd39aeaf8" Sep 08 18:56:30.925111 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.925090 2553 generic.go:356] "Generic (PLEG): container finished" podID="99bd63ad-bd5b-4e0b-ae93-70b20ab0310f" containerID="5603c09c604cc0b098ae045748ee994de193a22680f8e20d63dd97956a5e37bb" exitCode=0 Sep 08 18:56:30.925200 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.925134 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" Sep 08 18:56:30.925200 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.925139 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" event={"ID":"99bd63ad-bd5b-4e0b-ae93-70b20ab0310f","Type":"ContainerDied","Data":"5603c09c604cc0b098ae045748ee994de193a22680f8e20d63dd97956a5e37bb"} Sep 08 18:56:30.925200 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.925195 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw" event={"ID":"99bd63ad-bd5b-4e0b-ae93-70b20ab0310f","Type":"ContainerDied","Data":"302a6aaf989fb3aa3adcb150a4b78c87ba61449dce3a6b9c50aeff13b8880a44"} Sep 08 18:56:30.943502 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.943459 2553 patch_prober.go:28] interesting pod/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 18:56:30.943656 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.943528 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 18:56:30.959181 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.959158 2553 scope.go:117] "RemoveContainer" containerID="629d108319c2aebac5eb1451e50b12ac1d756a43f4fc53ea13a07a4cd39aeaf8" Sep 08 18:56:30.959500 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:56:30.959483 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629d108319c2aebac5eb1451e50b12ac1d756a43f4fc53ea13a07a4cd39aeaf8\": container with ID starting with 629d108319c2aebac5eb1451e50b12ac1d756a43f4fc53ea13a07a4cd39aeaf8 not found: ID does not exist" containerID="629d108319c2aebac5eb1451e50b12ac1d756a43f4fc53ea13a07a4cd39aeaf8" Sep 08 18:56:30.959595 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.959519 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629d108319c2aebac5eb1451e50b12ac1d756a43f4fc53ea13a07a4cd39aeaf8"} err="failed to get container status \"629d108319c2aebac5eb1451e50b12ac1d756a43f4fc53ea13a07a4cd39aeaf8\": rpc error: code = NotFound desc = could not find container \"629d108319c2aebac5eb1451e50b12ac1d756a43f4fc53ea13a07a4cd39aeaf8\": container with ID starting with 629d108319c2aebac5eb1451e50b12ac1d756a43f4fc53ea13a07a4cd39aeaf8 not found: ID does not exist" Sep 08 18:56:30.959595 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.959554 2553 scope.go:117] "RemoveContainer" containerID="5603c09c604cc0b098ae045748ee994de193a22680f8e20d63dd97956a5e37bb" Sep 08 18:56:30.968899 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.968882 2553 scope.go:117] "RemoveContainer" containerID="5603c09c604cc0b098ae045748ee994de193a22680f8e20d63dd97956a5e37bb" Sep 08 18:56:30.969183 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:56:30.969164 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5603c09c604cc0b098ae045748ee994de193a22680f8e20d63dd97956a5e37bb\": container with ID starting with 5603c09c604cc0b098ae045748ee994de193a22680f8e20d63dd97956a5e37bb not found: ID does not exist" containerID="5603c09c604cc0b098ae045748ee994de193a22680f8e20d63dd97956a5e37bb" Sep 08 18:56:30.969247 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.969194 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5603c09c604cc0b098ae045748ee994de193a22680f8e20d63dd97956a5e37bb"} err="failed to get container status \"5603c09c604cc0b098ae045748ee994de193a22680f8e20d63dd97956a5e37bb\": rpc error: code = NotFound desc = could not find container \"5603c09c604cc0b098ae045748ee994de193a22680f8e20d63dd97956a5e37bb\": container with ID starting with 5603c09c604cc0b098ae045748ee994de193a22680f8e20d63dd97956a5e37bb not found: ID does not exist" Sep 08 18:56:30.970946 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.970923 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w"] Sep 08 18:56:30.974149 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.974129 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52ba8b-6f8f474677-8lj4w"] Sep 08 18:56:30.984568 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.984546 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw"] Sep 08 18:56:30.988988 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:30.988967 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-52ba8b-b99f884c4-8fwzw"] Sep 08 18:56:31.939268 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:31.939226 2553 patch_prober.go:28] interesting pod/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 18:56:31.939785 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:31.939299 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 18:56:32.750063 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:32.750020 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b2c3088-389c-4794-b7cd-aedcf837398e" path="/var/lib/kubelet/pods/4b2c3088-389c-4794-b7cd-aedcf837398e/volumes" Sep 08 18:56:32.750621 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:32.750596 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99bd63ad-bd5b-4e0b-ae93-70b20ab0310f" path="/var/lib/kubelet/pods/99bd63ad-bd5b-4e0b-ae93-70b20ab0310f/volumes" Sep 08 18:56:32.939081 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:32.939045 2553 patch_prober.go:28] interesting pod/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 18:56:32.939489 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:32.939461 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 18:56:33.939816 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:33.939775 2553 patch_prober.go:28] interesting pod/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 18:56:33.940311 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:33.939850 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 18:56:34.939232 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:34.939196 2553 patch_prober.go:28] interesting pod/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 08 18:56:34.939562 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:34.939520 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 08 18:56:35.943032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:35.942996 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:35.943460 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:35.943286 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:35.944054 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:35.944036 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:56:37.651508 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.651468 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt"] Sep 08 18:56:37.652016 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.651996 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99bd63ad-bd5b-4e0b-ae93-70b20ab0310f" containerName="payload-pre-processing" Sep 08 18:56:37.652109 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.652019 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="99bd63ad-bd5b-4e0b-ae93-70b20ab0310f" containerName="payload-pre-processing" Sep 08 18:56:37.652109 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.652042 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2c3088-389c-4794-b7cd-aedcf837398e" containerName="payload-processing" Sep 08 18:56:37.652109 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.652052 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2c3088-389c-4794-b7cd-aedcf837398e" containerName="payload-processing" Sep 08 18:56:37.652109 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.652065 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48fa7b97-3831-462a-84af-8cc87f2d2de6" containerName="revoke-keys" Sep 08 18:56:37.652109 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.652075 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="48fa7b97-3831-462a-84af-8cc87f2d2de6" containerName="revoke-keys" Sep 08 18:56:37.652622 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.652178 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="48fa7b97-3831-462a-84af-8cc87f2d2de6" containerName="revoke-keys" Sep 08 18:56:37.652622 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.652452 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b2c3088-389c-4794-b7cd-aedcf837398e" containerName="payload-processing" Sep 08 18:56:37.652622 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.652474 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="99bd63ad-bd5b-4e0b-ae93-70b20ab0310f" containerName="payload-pre-processing" Sep 08 18:56:37.656957 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.656936 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt" Sep 08 18:56:37.663946 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.663918 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 18:56:37.664054 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.663949 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 18:56:37.674336 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.674310 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt"] Sep 08 18:56:37.729485 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.729458 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53f9ea6-7a0d-4204-8973-c9887494e3fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt\" (UID: \"c53f9ea6-7a0d-4204-8973-c9887494e3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt" Sep 08 18:56:37.830485 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.830447 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53f9ea6-7a0d-4204-8973-c9887494e3fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt\" (UID: \"c53f9ea6-7a0d-4204-8973-c9887494e3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt" Sep 08 18:56:37.831066 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.831044 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53f9ea6-7a0d-4204-8973-c9887494e3fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt\" (UID: \"c53f9ea6-7a0d-4204-8973-c9887494e3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt" Sep 08 18:56:37.968582 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:37.968502 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt" Sep 08 18:56:38.240034 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:38.240005 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt"] Sep 08 18:56:38.240965 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:56:38.240941 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53f9ea6_7a0d_4204_8973_c9887494e3fb.slice/crio-ace2863567bfdbfc816bdc5457fbda2abad56e7bf56a83a1956b6dec70eb2052 WatchSource:0}: Error finding container ace2863567bfdbfc816bdc5457fbda2abad56e7bf56a83a1956b6dec70eb2052: Status 404 returned error can't find the container with id ace2863567bfdbfc816bdc5457fbda2abad56e7bf56a83a1956b6dec70eb2052 Sep 08 18:56:38.983843 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:38.983812 2553 generic.go:356] "Generic (PLEG): container finished" podID="c53f9ea6-7a0d-4204-8973-c9887494e3fb" containerID="9ce5d3e2b43ed5b27764547c1b9163f8a21f03bb1872e80f440f0b7d422f4800" exitCode=0 Sep 08 18:56:38.984163 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:38.983890 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt" event={"ID":"c53f9ea6-7a0d-4204-8973-c9887494e3fb","Type":"ContainerDied","Data":"9ce5d3e2b43ed5b27764547c1b9163f8a21f03bb1872e80f440f0b7d422f4800"} Sep 08 18:56:38.984163 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:38.983942 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt" event={"ID":"c53f9ea6-7a0d-4204-8973-c9887494e3fb","Type":"ContainerStarted","Data":"ace2863567bfdbfc816bdc5457fbda2abad56e7bf56a83a1956b6dec70eb2052"} Sep 08 18:56:40.126837 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:40.126296 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt" Sep 08 18:56:40.251841 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:40.251765 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53f9ea6-7a0d-4204-8973-c9887494e3fb-maas-api-service-ca\") pod \"c53f9ea6-7a0d-4204-8973-c9887494e3fb\" (UID: \"c53f9ea6-7a0d-4204-8973-c9887494e3fb\") " Sep 08 18:56:40.252136 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:40.252113 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53f9ea6-7a0d-4204-8973-c9887494e3fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c53f9ea6-7a0d-4204-8973-c9887494e3fb" (UID: "c53f9ea6-7a0d-4204-8973-c9887494e3fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:56:40.353281 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:40.353248 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c53f9ea6-7a0d-4204-8973-c9887494e3fb-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:56:40.996108 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:40.996072 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt" event={"ID":"c53f9ea6-7a0d-4204-8973-c9887494e3fb","Type":"ContainerDied","Data":"ace2863567bfdbfc816bdc5457fbda2abad56e7bf56a83a1956b6dec70eb2052"} Sep 08 18:56:40.996278 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:40.996119 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ace2863567bfdbfc816bdc5457fbda2abad56e7bf56a83a1956b6dec70eb2052" Sep 08 18:56:40.996278 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:40.996086 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt" Sep 08 18:56:44.250437 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:44.250399 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-64effff4-5rmgt"] Sep 08 18:56:44.250988 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:44.250968 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c53f9ea6-7a0d-4204-8973-c9887494e3fb" containerName="revoke-keys" Sep 08 18:56:44.251079 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:44.250991 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c53f9ea6-7a0d-4204-8973-c9887494e3fb" containerName="revoke-keys" Sep 08 18:56:44.251134 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:44.251083 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c53f9ea6-7a0d-4204-8973-c9887494e3fb" containerName="revoke-keys" Sep 08 18:56:44.254390 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:44.254295 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-64effff4-5rmgt" Sep 08 18:56:44.257870 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:44.257850 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 18:56:44.258124 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:44.258105 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 18:56:44.265795 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:44.265772 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-64effff4-5rmgt"] Sep 08 18:56:44.388869 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:44.388832 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b93b4a-671c-472e-b4c5-17515668638e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-64effff4-5rmgt\" (UID: \"c7b93b4a-671c-472e-b4c5-17515668638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-64effff4-5rmgt" Sep 08 18:56:44.489645 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:44.489614 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b93b4a-671c-472e-b4c5-17515668638e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-64effff4-5rmgt\" (UID: \"c7b93b4a-671c-472e-b4c5-17515668638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-64effff4-5rmgt" Sep 08 18:56:44.490205 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:44.490184 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b93b4a-671c-472e-b4c5-17515668638e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-64effff4-5rmgt\" (UID: \"c7b93b4a-671c-472e-b4c5-17515668638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-64effff4-5rmgt" Sep 08 18:56:44.568020 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:44.567948 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-64effff4-5rmgt" Sep 08 18:56:44.842963 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:44.842936 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-64effff4-5rmgt"] Sep 08 18:56:44.844630 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:56:44.844603 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b93b4a_671c_472e_b4c5_17515668638e.slice/crio-daf2c847ebdc1b0198c6b5634ef127f316f0bc7fea78566ddae47ad3be5fb328 WatchSource:0}: Error finding container daf2c847ebdc1b0198c6b5634ef127f316f0bc7fea78566ddae47ad3be5fb328: Status 404 returned error can't find the container with id daf2c847ebdc1b0198c6b5634ef127f316f0bc7fea78566ddae47ad3be5fb328 Sep 08 18:56:45.024661 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:45.024626 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-64effff4-5rmgt" event={"ID":"c7b93b4a-671c-472e-b4c5-17515668638e","Type":"ContainerStarted","Data":"daf2c847ebdc1b0198c6b5634ef127f316f0bc7fea78566ddae47ad3be5fb328"} Sep 08 18:56:46.031880 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:46.031843 2553 generic.go:356] "Generic (PLEG): container finished" podID="c7b93b4a-671c-472e-b4c5-17515668638e" containerID="6cf31c058700fddd8b5788059cee06d8b28cdf79681766ad0c6ae1a67eb61d3a" exitCode=0 Sep 08 18:56:46.032246 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:46.031929 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-64effff4-5rmgt" event={"ID":"c7b93b4a-671c-472e-b4c5-17515668638e","Type":"ContainerDied","Data":"6cf31c058700fddd8b5788059cee06d8b28cdf79681766ad0c6ae1a67eb61d3a"} Sep 08 18:56:47.170088 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:47.170065 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-64effff4-5rmgt" Sep 08 18:56:47.317223 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:47.317138 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b93b4a-671c-472e-b4c5-17515668638e-maas-api-service-ca\") pod \"c7b93b4a-671c-472e-b4c5-17515668638e\" (UID: \"c7b93b4a-671c-472e-b4c5-17515668638e\") " Sep 08 18:56:47.317521 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:47.317498 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b93b4a-671c-472e-b4c5-17515668638e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c7b93b4a-671c-472e-b4c5-17515668638e" (UID: "c7b93b4a-671c-472e-b4c5-17515668638e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:56:47.418172 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:47.418143 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7b93b4a-671c-472e-b4c5-17515668638e-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:56:48.045689 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.045660 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-64effff4-5rmgt" Sep 08 18:56:48.045882 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.045699 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-64effff4-5rmgt" event={"ID":"c7b93b4a-671c-472e-b4c5-17515668638e","Type":"ContainerDied","Data":"daf2c847ebdc1b0198c6b5634ef127f316f0bc7fea78566ddae47ad3be5fb328"} Sep 08 18:56:48.045882 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.045759 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daf2c847ebdc1b0198c6b5634ef127f316f0bc7fea78566ddae47ad3be5fb328" Sep 08 18:56:48.399865 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.399829 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j"] Sep 08 18:56:48.400925 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.400886 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" podUID="643a9a33-2c05-47a9-9bd3-fb58abfa8705" containerName="payload-processing" containerID="cri-o://cff8c08e9ab1fb3a8e7fa7cf6aee41fe1d8e2f43508e0efc5d16f13e2a7c621c" gracePeriod=30 Sep 08 18:56:48.405180 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.404792 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44"] Sep 08 18:56:48.405180 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.404991 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" podUID="834b0555-3f80-49de-a999-93e66bad8521" containerName="payload-pre-processing" containerID="cri-o://e0503a20e8735d484aff4889e2b39f3e066aec22a6aa904d8f6d08f65541d1e7" gracePeriod=30 Sep 08 18:56:48.547058 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.546975 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt"] Sep 08 18:56:48.549230 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.549207 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9c3074-mlpvt"] Sep 08 18:56:48.692431 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.692398 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" Sep 08 18:56:48.695432 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.695411 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" Sep 08 18:56:48.746633 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.746595 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c53f9ea6-7a0d-4204-8973-c9887494e3fb" path="/var/lib/kubelet/pods/c53f9ea6-7a0d-4204-8973-c9887494e3fb/volumes" Sep 08 18:56:48.832026 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.831996 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643a9a33-2c05-47a9-9bd3-fb58abfa8705-plugins-config-volume\") pod \"643a9a33-2c05-47a9-9bd3-fb58abfa8705\" (UID: \"643a9a33-2c05-47a9-9bd3-fb58abfa8705\") " Sep 08 18:56:48.832195 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.832061 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htf86\" (UniqueName: \"kubernetes.io/projected/643a9a33-2c05-47a9-9bd3-fb58abfa8705-kube-api-access-htf86\") pod \"643a9a33-2c05-47a9-9bd3-fb58abfa8705\" (UID: \"643a9a33-2c05-47a9-9bd3-fb58abfa8705\") " Sep 08 18:56:48.832195 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.832132 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834b0555-3f80-49de-a999-93e66bad8521-plugins-config-volume\") pod \"834b0555-3f80-49de-a999-93e66bad8521\" (UID: \"834b0555-3f80-49de-a999-93e66bad8521\") " Sep 08 18:56:48.832195 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.832157 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdl86\" (UniqueName: \"kubernetes.io/projected/834b0555-3f80-49de-a999-93e66bad8521-kube-api-access-jdl86\") pod \"834b0555-3f80-49de-a999-93e66bad8521\" (UID: \"834b0555-3f80-49de-a999-93e66bad8521\") " Sep 08 18:56:48.832701 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.832451 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643a9a33-2c05-47a9-9bd3-fb58abfa8705-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "643a9a33-2c05-47a9-9bd3-fb58abfa8705" (UID: "643a9a33-2c05-47a9-9bd3-fb58abfa8705"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:56:48.832701 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.832546 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/643a9a33-2c05-47a9-9bd3-fb58abfa8705-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:56:48.832701 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.832588 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/834b0555-3f80-49de-a999-93e66bad8521-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "834b0555-3f80-49de-a999-93e66bad8521" (UID: "834b0555-3f80-49de-a999-93e66bad8521"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:56:48.834227 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.834207 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643a9a33-2c05-47a9-9bd3-fb58abfa8705-kube-api-access-htf86" (OuterVolumeSpecName: "kube-api-access-htf86") pod "643a9a33-2c05-47a9-9bd3-fb58abfa8705" (UID: "643a9a33-2c05-47a9-9bd3-fb58abfa8705"). InnerVolumeSpecName "kube-api-access-htf86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:56:48.834286 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.834230 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834b0555-3f80-49de-a999-93e66bad8521-kube-api-access-jdl86" (OuterVolumeSpecName: "kube-api-access-jdl86") pod "834b0555-3f80-49de-a999-93e66bad8521" (UID: "834b0555-3f80-49de-a999-93e66bad8521"). InnerVolumeSpecName "kube-api-access-jdl86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:56:48.933653 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.933587 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htf86\" (UniqueName: \"kubernetes.io/projected/643a9a33-2c05-47a9-9bd3-fb58abfa8705-kube-api-access-htf86\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:56:48.933653 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.933611 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/834b0555-3f80-49de-a999-93e66bad8521-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:56:48.933653 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:48.933621 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdl86\" (UniqueName: \"kubernetes.io/projected/834b0555-3f80-49de-a999-93e66bad8521-kube-api-access-jdl86\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:56:49.060099 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.060065 2553 generic.go:356] "Generic (PLEG): container finished" podID="834b0555-3f80-49de-a999-93e66bad8521" containerID="e0503a20e8735d484aff4889e2b39f3e066aec22a6aa904d8f6d08f65541d1e7" exitCode=0 Sep 08 18:56:49.060294 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.060117 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" Sep 08 18:56:49.060294 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.060148 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" event={"ID":"834b0555-3f80-49de-a999-93e66bad8521","Type":"ContainerDied","Data":"e0503a20e8735d484aff4889e2b39f3e066aec22a6aa904d8f6d08f65541d1e7"} Sep 08 18:56:49.060294 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.060236 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44" event={"ID":"834b0555-3f80-49de-a999-93e66bad8521","Type":"ContainerDied","Data":"5818aff4805315219f39e616ef41a077d6458e00201b4e72d6cd60d5e928ea34"} Sep 08 18:56:49.060294 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.060264 2553 scope.go:117] "RemoveContainer" containerID="e0503a20e8735d484aff4889e2b39f3e066aec22a6aa904d8f6d08f65541d1e7" Sep 08 18:56:49.064742 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.064716 2553 generic.go:356] "Generic (PLEG): container finished" podID="643a9a33-2c05-47a9-9bd3-fb58abfa8705" containerID="cff8c08e9ab1fb3a8e7fa7cf6aee41fe1d8e2f43508e0efc5d16f13e2a7c621c" exitCode=0 Sep 08 18:56:49.064980 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.064807 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" event={"ID":"643a9a33-2c05-47a9-9bd3-fb58abfa8705","Type":"ContainerDied","Data":"cff8c08e9ab1fb3a8e7fa7cf6aee41fe1d8e2f43508e0efc5d16f13e2a7c621c"} Sep 08 18:56:49.065117 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.065078 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" event={"ID":"643a9a33-2c05-47a9-9bd3-fb58abfa8705","Type":"ContainerDied","Data":"8834a462eab7b1b794d0ad00bec3b6c8692629a6ecc01ddbf8211c529749ebc6"} Sep 08 18:56:49.065231 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.064914 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j" Sep 08 18:56:49.072667 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.072649 2553 scope.go:117] "RemoveContainer" containerID="e0503a20e8735d484aff4889e2b39f3e066aec22a6aa904d8f6d08f65541d1e7" Sep 08 18:56:49.072945 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:56:49.072926 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0503a20e8735d484aff4889e2b39f3e066aec22a6aa904d8f6d08f65541d1e7\": container with ID starting with e0503a20e8735d484aff4889e2b39f3e066aec22a6aa904d8f6d08f65541d1e7 not found: ID does not exist" containerID="e0503a20e8735d484aff4889e2b39f3e066aec22a6aa904d8f6d08f65541d1e7" Sep 08 18:56:49.072995 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.072953 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0503a20e8735d484aff4889e2b39f3e066aec22a6aa904d8f6d08f65541d1e7"} err="failed to get container status \"e0503a20e8735d484aff4889e2b39f3e066aec22a6aa904d8f6d08f65541d1e7\": rpc error: code = NotFound desc = could not find container \"e0503a20e8735d484aff4889e2b39f3e066aec22a6aa904d8f6d08f65541d1e7\": container with ID starting with e0503a20e8735d484aff4889e2b39f3e066aec22a6aa904d8f6d08f65541d1e7 not found: ID does not exist" Sep 08 18:56:49.072995 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.072968 2553 scope.go:117] "RemoveContainer" containerID="cff8c08e9ab1fb3a8e7fa7cf6aee41fe1d8e2f43508e0efc5d16f13e2a7c621c" Sep 08 18:56:49.081658 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.081619 2553 scope.go:117] "RemoveContainer" containerID="cff8c08e9ab1fb3a8e7fa7cf6aee41fe1d8e2f43508e0efc5d16f13e2a7c621c" Sep 08 18:56:49.082115 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:56:49.082008 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff8c08e9ab1fb3a8e7fa7cf6aee41fe1d8e2f43508e0efc5d16f13e2a7c621c\": container with ID starting with cff8c08e9ab1fb3a8e7fa7cf6aee41fe1d8e2f43508e0efc5d16f13e2a7c621c not found: ID does not exist" containerID="cff8c08e9ab1fb3a8e7fa7cf6aee41fe1d8e2f43508e0efc5d16f13e2a7c621c" Sep 08 18:56:49.082115 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.082041 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff8c08e9ab1fb3a8e7fa7cf6aee41fe1d8e2f43508e0efc5d16f13e2a7c621c"} err="failed to get container status \"cff8c08e9ab1fb3a8e7fa7cf6aee41fe1d8e2f43508e0efc5d16f13e2a7c621c\": rpc error: code = NotFound desc = could not find container \"cff8c08e9ab1fb3a8e7fa7cf6aee41fe1d8e2f43508e0efc5d16f13e2a7c621c\": container with ID starting with cff8c08e9ab1fb3a8e7fa7cf6aee41fe1d8e2f43508e0efc5d16f13e2a7c621c not found: ID does not exist" Sep 08 18:56:49.084326 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.084300 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44"] Sep 08 18:56:49.088096 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.088075 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c3074-d8c48c786-jlg44"] Sep 08 18:56:49.100349 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.100329 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j"] Sep 08 18:56:49.106246 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:49.106226 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c3074-5cb96bf56c-n459j"] Sep 08 18:56:50.744655 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:50.744622 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="643a9a33-2c05-47a9-9bd3-fb58abfa8705" path="/var/lib/kubelet/pods/643a9a33-2c05-47a9-9bd3-fb58abfa8705/volumes" Sep 08 18:56:50.745000 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:50.744981 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="834b0555-3f80-49de-a999-93e66bad8521" path="/var/lib/kubelet/pods/834b0555-3f80-49de-a999-93e66bad8521/volumes" Sep 08 18:56:55.265932 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:55.265900 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-64effff4-5rmgt"] Sep 08 18:56:55.268657 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:55.268629 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-64effff4-5rmgt"] Sep 08 18:56:56.746793 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:56:56.746752 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7b93b4a-671c-472e-b4c5-17515668638e" path="/var/lib/kubelet/pods/c7b93b4a-671c-472e-b4c5-17515668638e/volumes" Sep 08 18:57:01.059430 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.059394 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm"] Sep 08 18:57:01.059940 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.059771 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834b0555-3f80-49de-a999-93e66bad8521" containerName="payload-pre-processing" Sep 08 18:57:01.059940 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.059787 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="834b0555-3f80-49de-a999-93e66bad8521" containerName="payload-pre-processing" Sep 08 18:57:01.059940 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.059810 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643a9a33-2c05-47a9-9bd3-fb58abfa8705" containerName="payload-processing" Sep 08 18:57:01.059940 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.059820 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="643a9a33-2c05-47a9-9bd3-fb58abfa8705" containerName="payload-processing" Sep 08 18:57:01.059940 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.059833 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b93b4a-671c-472e-b4c5-17515668638e" containerName="revoke-keys" Sep 08 18:57:01.059940 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.059839 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b93b4a-671c-472e-b4c5-17515668638e" containerName="revoke-keys" Sep 08 18:57:01.060272 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.059949 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b93b4a-671c-472e-b4c5-17515668638e" containerName="revoke-keys" Sep 08 18:57:01.060272 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.059968 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="834b0555-3f80-49de-a999-93e66bad8521" containerName="payload-pre-processing" Sep 08 18:57:01.060272 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.059977 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="643a9a33-2c05-47a9-9bd3-fb58abfa8705" containerName="payload-processing" Sep 08 18:57:01.063257 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.063240 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" Sep 08 18:57:01.065925 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.065902 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-a4001b-dockercfg-h82r6\"" Sep 08 18:57:01.066073 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.066054 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-a4001b\"" Sep 08 18:57:01.074625 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.074604 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm"] Sep 08 18:57:01.241818 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.241779 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qflv8\" (UniqueName: \"kubernetes.io/projected/00659bb0-6e3b-4f6a-8816-58abc54f85c9-kube-api-access-qflv8\") pod \"payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm\" (UID: \"00659bb0-6e3b-4f6a-8816-58abc54f85c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" Sep 08 18:57:01.241978 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.241858 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00659bb0-6e3b-4f6a-8816-58abc54f85c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm\" (UID: \"00659bb0-6e3b-4f6a-8816-58abc54f85c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" Sep 08 18:57:01.343058 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.342980 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00659bb0-6e3b-4f6a-8816-58abc54f85c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm\" (UID: \"00659bb0-6e3b-4f6a-8816-58abc54f85c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" Sep 08 18:57:01.343201 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.343058 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qflv8\" (UniqueName: \"kubernetes.io/projected/00659bb0-6e3b-4f6a-8816-58abc54f85c9-kube-api-access-qflv8\") pod \"payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm\" (UID: \"00659bb0-6e3b-4f6a-8816-58abc54f85c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" Sep 08 18:57:01.343678 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.343659 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00659bb0-6e3b-4f6a-8816-58abc54f85c9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm\" (UID: \"00659bb0-6e3b-4f6a-8816-58abc54f85c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" Sep 08 18:57:01.354100 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.354074 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qflv8\" (UniqueName: \"kubernetes.io/projected/00659bb0-6e3b-4f6a-8816-58abc54f85c9-kube-api-access-qflv8\") pod \"payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm\" (UID: \"00659bb0-6e3b-4f6a-8816-58abc54f85c9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" Sep 08 18:57:01.373939 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.373914 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" Sep 08 18:57:01.644858 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:01.644832 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm"] Sep 08 18:57:01.647102 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:57:01.647071 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00659bb0_6e3b_4f6a_8816_58abc54f85c9.slice/crio-bd4274e156e824b7ddb2abbc2d33406afe994203a429a14fe3365f94007f73a5 WatchSource:0}: Error finding container bd4274e156e824b7ddb2abbc2d33406afe994203a429a14fe3365f94007f73a5: Status 404 returned error can't find the container with id bd4274e156e824b7ddb2abbc2d33406afe994203a429a14fe3365f94007f73a5 Sep 08 18:57:02.146750 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:02.146705 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" event={"ID":"00659bb0-6e3b-4f6a-8816-58abc54f85c9","Type":"ContainerStarted","Data":"6b63a60755d4d703457c1a018229e691d037eabeba579b67e11c9a8eb240a499"} Sep 08 18:57:02.147241 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:02.146772 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" event={"ID":"00659bb0-6e3b-4f6a-8816-58abc54f85c9","Type":"ContainerStarted","Data":"bd4274e156e824b7ddb2abbc2d33406afe994203a429a14fe3365f94007f73a5"} Sep 08 18:57:02.147241 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:02.146799 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" Sep 08 18:57:02.171547 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:02.171489 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" podStartSLOduration=1.171466331 podStartE2EDuration="1.171466331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:02.167676607 +0000 UTC m=+1357.873564870" watchObservedRunningTime="2026-09-08 18:57:02.171466331 +0000 UTC m=+1357.877354598" Sep 08 18:57:02.316629 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:02.316589 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s"] Sep 08 18:57:02.316824 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:02.316797 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerName="istio-proxy" containerID="cri-o://cd5089be0fe6f7a91b84287d7f0b68ec79c2afac1284d7436e3dbafaf18ad711" gracePeriod=30 Sep 08 18:57:05.945410 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:05.945352 2553 patch_prober.go:28] interesting pod/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 18:57:05.945855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:05.945436 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 18:57:07.569889 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.569863 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:57:07.605219 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.605153 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-podinfo\") pod \"18d4a538-ab50-4491-a6ed-d8edea7dc906\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " Sep 08 18:57:07.605219 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.605194 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-data\") pod \"18d4a538-ab50-4491-a6ed-d8edea7dc906\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " Sep 08 18:57:07.605442 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.605238 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-workload-certs\") pod \"18d4a538-ab50-4491-a6ed-d8edea7dc906\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " Sep 08 18:57:07.605442 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.605260 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-envoy\") pod \"18d4a538-ab50-4491-a6ed-d8edea7dc906\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " Sep 08 18:57:07.605442 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.605309 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-workload-socket\") pod \"18d4a538-ab50-4491-a6ed-d8edea7dc906\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " Sep 08 18:57:07.605442 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.605333 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-token\") pod \"18d4a538-ab50-4491-a6ed-d8edea7dc906\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " Sep 08 18:57:07.605442 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.605418 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-credential-socket\") pod \"18d4a538-ab50-4491-a6ed-d8edea7dc906\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " Sep 08 18:57:07.605703 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.605449 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d4a538-ab50-4491-a6ed-d8edea7dc906-istiod-ca-cert\") pod \"18d4a538-ab50-4491-a6ed-d8edea7dc906\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " Sep 08 18:57:07.605703 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.605500 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgv54\" (UniqueName: \"kubernetes.io/projected/18d4a538-ab50-4491-a6ed-d8edea7dc906-kube-api-access-lgv54\") pod \"18d4a538-ab50-4491-a6ed-d8edea7dc906\" (UID: \"18d4a538-ab50-4491-a6ed-d8edea7dc906\") " Sep 08 18:57:07.605703 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.605582 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18d4a538-ab50-4491-a6ed-d8edea7dc906" (UID: "18d4a538-ab50-4491-a6ed-d8edea7dc906"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:57:07.605862 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.605718 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-data" (OuterVolumeSpecName: "istio-data") pod "18d4a538-ab50-4491-a6ed-d8edea7dc906" (UID: "18d4a538-ab50-4491-a6ed-d8edea7dc906"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:57:07.605932 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.605911 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18d4a538-ab50-4491-a6ed-d8edea7dc906" (UID: "18d4a538-ab50-4491-a6ed-d8edea7dc906"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:57:07.605986 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.605851 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:57:07.605986 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.605968 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:57:07.605986 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.605932 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18d4a538-ab50-4491-a6ed-d8edea7dc906" (UID: "18d4a538-ab50-4491-a6ed-d8edea7dc906"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:57:07.606144 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.606005 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d4a538-ab50-4491-a6ed-d8edea7dc906-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18d4a538-ab50-4491-a6ed-d8edea7dc906" (UID: "18d4a538-ab50-4491-a6ed-d8edea7dc906"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:57:07.607707 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.607683 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d4a538-ab50-4491-a6ed-d8edea7dc906-kube-api-access-lgv54" (OuterVolumeSpecName: "kube-api-access-lgv54") pod "18d4a538-ab50-4491-a6ed-d8edea7dc906" (UID: "18d4a538-ab50-4491-a6ed-d8edea7dc906"). InnerVolumeSpecName "kube-api-access-lgv54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:57:07.607707 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.607682 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18d4a538-ab50-4491-a6ed-d8edea7dc906" (UID: "18d4a538-ab50-4491-a6ed-d8edea7dc906"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 18:57:07.607939 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.607911 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18d4a538-ab50-4491-a6ed-d8edea7dc906" (UID: "18d4a538-ab50-4491-a6ed-d8edea7dc906"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 18:57:07.607939 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.607922 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-token" (OuterVolumeSpecName: "istio-token") pod "18d4a538-ab50-4491-a6ed-d8edea7dc906" (UID: "18d4a538-ab50-4491-a6ed-d8edea7dc906"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 18:57:07.706925 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.706891 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:57:07.706925 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.706920 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:57:07.706925 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.706929 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:57:07.707149 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.706938 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d4a538-ab50-4491-a6ed-d8edea7dc906-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:57:07.707149 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.706948 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgv54\" (UniqueName: \"kubernetes.io/projected/18d4a538-ab50-4491-a6ed-d8edea7dc906-kube-api-access-lgv54\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:57:07.707149 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.706957 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:57:07.707149 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:07.706965 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d4a538-ab50-4491-a6ed-d8edea7dc906-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:57:08.183336 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:08.183303 2553 generic.go:356] "Generic (PLEG): container finished" podID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerID="cd5089be0fe6f7a91b84287d7f0b68ec79c2afac1284d7436e3dbafaf18ad711" exitCode=0 Sep 08 18:57:08.183496 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:08.183341 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" Sep 08 18:57:08.183496 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:08.183391 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" event={"ID":"18d4a538-ab50-4491-a6ed-d8edea7dc906","Type":"ContainerDied","Data":"cd5089be0fe6f7a91b84287d7f0b68ec79c2afac1284d7436e3dbafaf18ad711"} Sep 08 18:57:08.183496 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:08.183439 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s" event={"ID":"18d4a538-ab50-4491-a6ed-d8edea7dc906","Type":"ContainerDied","Data":"d8933d4a11cbfafcff2d1e4fce708dbdfb7d49f76dde84ce619a4fbbdaa64d7f"} Sep 08 18:57:08.183496 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:08.183458 2553 scope.go:117] "RemoveContainer" containerID="cd5089be0fe6f7a91b84287d7f0b68ec79c2afac1284d7436e3dbafaf18ad711" Sep 08 18:57:08.195131 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:08.195105 2553 scope.go:117] "RemoveContainer" containerID="518a6d7895e12f3676b4c20630ff20d00036128a4cc5e65e45294e0f97eeff9e" Sep 08 18:57:08.204590 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:08.204571 2553 scope.go:117] "RemoveContainer" containerID="cd5089be0fe6f7a91b84287d7f0b68ec79c2afac1284d7436e3dbafaf18ad711" Sep 08 18:57:08.204871 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:57:08.204847 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5089be0fe6f7a91b84287d7f0b68ec79c2afac1284d7436e3dbafaf18ad711\": container with ID starting with cd5089be0fe6f7a91b84287d7f0b68ec79c2afac1284d7436e3dbafaf18ad711 not found: ID does not exist" containerID="cd5089be0fe6f7a91b84287d7f0b68ec79c2afac1284d7436e3dbafaf18ad711" Sep 08 18:57:08.204973 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:08.204877 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5089be0fe6f7a91b84287d7f0b68ec79c2afac1284d7436e3dbafaf18ad711"} err="failed to get container status \"cd5089be0fe6f7a91b84287d7f0b68ec79c2afac1284d7436e3dbafaf18ad711\": rpc error: code = NotFound desc = could not find container \"cd5089be0fe6f7a91b84287d7f0b68ec79c2afac1284d7436e3dbafaf18ad711\": container with ID starting with cd5089be0fe6f7a91b84287d7f0b68ec79c2afac1284d7436e3dbafaf18ad711 not found: ID does not exist" Sep 08 18:57:08.204973 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:08.204896 2553 scope.go:117] "RemoveContainer" containerID="518a6d7895e12f3676b4c20630ff20d00036128a4cc5e65e45294e0f97eeff9e" Sep 08 18:57:08.205172 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:57:08.205155 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518a6d7895e12f3676b4c20630ff20d00036128a4cc5e65e45294e0f97eeff9e\": container with ID starting with 518a6d7895e12f3676b4c20630ff20d00036128a4cc5e65e45294e0f97eeff9e not found: ID does not exist" containerID="518a6d7895e12f3676b4c20630ff20d00036128a4cc5e65e45294e0f97eeff9e" Sep 08 18:57:08.205219 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:08.205175 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518a6d7895e12f3676b4c20630ff20d00036128a4cc5e65e45294e0f97eeff9e"} err="failed to get container status \"518a6d7895e12f3676b4c20630ff20d00036128a4cc5e65e45294e0f97eeff9e\": rpc error: code = NotFound desc = could not find container \"518a6d7895e12f3676b4c20630ff20d00036128a4cc5e65e45294e0f97eeff9e\": container with ID starting with 518a6d7895e12f3676b4c20630ff20d00036128a4cc5e65e45294e0f97eeff9e not found: ID does not exist" Sep 08 18:57:08.206642 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:08.206581 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s"] Sep 08 18:57:08.211178 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:08.211154 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-64effff4-openshift-default-6fc8c75fd8-c8g7s"] Sep 08 18:57:08.219461 ip-10-0-132-43 kubenswrapper[2553]: E0908 18:57:08.219432 2553 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d4a538_ab50_4491_a6ed_d8edea7dc906.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d4a538_ab50_4491_a6ed_d8edea7dc906.slice/crio-d8933d4a11cbfafcff2d1e4fce708dbdfb7d49f76dde84ce619a4fbbdaa64d7f\": RecentStats: unable to find data in memory cache]" Sep 08 18:57:08.744898 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:08.744861 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" path="/var/lib/kubelet/pods/18d4a538-ab50-4491-a6ed-d8edea7dc906/volumes" Sep 08 18:57:13.156494 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:13.156457 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" Sep 08 18:57:21.085098 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.085055 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5"] Sep 08 18:57:21.085637 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.085617 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerName="istio-proxy" Sep 08 18:57:21.085722 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.085640 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerName="istio-proxy" Sep 08 18:57:21.085722 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.085660 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerName="istio-proxy" Sep 08 18:57:21.085722 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.085668 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerName="istio-proxy" Sep 08 18:57:21.085875 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.085757 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerName="istio-proxy" Sep 08 18:57:21.085875 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.085772 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="18d4a538-ab50-4491-a6ed-d8edea7dc906" containerName="istio-proxy" Sep 08 18:57:21.090266 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.090244 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" Sep 08 18:57:21.093637 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.093611 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-093570-dockercfg-z2fh8\"" Sep 08 18:57:21.093741 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.093640 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-093570\"" Sep 08 18:57:21.102688 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.102666 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5"] Sep 08 18:57:21.123028 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.123004 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhzmc\" (UniqueName: \"kubernetes.io/projected/a9588f03-8418-4ff4-8001-743797e1cd5a-kube-api-access-hhzmc\") pod \"payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5\" (UID: \"a9588f03-8418-4ff4-8001-743797e1cd5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" Sep 08 18:57:21.123127 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.123061 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9588f03-8418-4ff4-8001-743797e1cd5a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5\" (UID: \"a9588f03-8418-4ff4-8001-743797e1cd5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" Sep 08 18:57:21.224289 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.224241 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzmc\" (UniqueName: \"kubernetes.io/projected/a9588f03-8418-4ff4-8001-743797e1cd5a-kube-api-access-hhzmc\") pod \"payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5\" (UID: \"a9588f03-8418-4ff4-8001-743797e1cd5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" Sep 08 18:57:21.224525 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.224350 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9588f03-8418-4ff4-8001-743797e1cd5a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5\" (UID: \"a9588f03-8418-4ff4-8001-743797e1cd5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" Sep 08 18:57:21.225323 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.225299 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9588f03-8418-4ff4-8001-743797e1cd5a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5\" (UID: \"a9588f03-8418-4ff4-8001-743797e1cd5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" Sep 08 18:57:21.233931 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.233906 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzmc\" (UniqueName: \"kubernetes.io/projected/a9588f03-8418-4ff4-8001-743797e1cd5a-kube-api-access-hhzmc\") pod \"payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5\" (UID: \"a9588f03-8418-4ff4-8001-743797e1cd5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" Sep 08 18:57:21.401046 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.401015 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" Sep 08 18:57:21.669570 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:21.669536 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5"] Sep 08 18:57:21.670640 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:57:21.670611 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9588f03_8418_4ff4_8001_743797e1cd5a.slice/crio-8abb6f570d4942e1ae71d4b0908687c865ce60fc74134e500b9fdc90e7144f09 WatchSource:0}: Error finding container 8abb6f570d4942e1ae71d4b0908687c865ce60fc74134e500b9fdc90e7144f09: Status 404 returned error can't find the container with id 8abb6f570d4942e1ae71d4b0908687c865ce60fc74134e500b9fdc90e7144f09 Sep 08 18:57:22.268416 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:22.268374 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" event={"ID":"a9588f03-8418-4ff4-8001-743797e1cd5a","Type":"ContainerStarted","Data":"759338af8d0e7a9c38be6d7bf8ee35243d2508920104117f0f9fbc5fb042e556"} Sep 08 18:57:22.268791 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:22.268446 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" Sep 08 18:57:22.268791 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:22.268471 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" event={"ID":"a9588f03-8418-4ff4-8001-743797e1cd5a","Type":"ContainerStarted","Data":"8abb6f570d4942e1ae71d4b0908687c865ce60fc74134e500b9fdc90e7144f09"} Sep 08 18:57:22.289007 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:22.288956 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" podStartSLOduration=1.288939931 podStartE2EDuration="1.288939931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:22.285779196 +0000 UTC m=+1377.991667428" watchObservedRunningTime="2026-09-08 18:57:22.288939931 +0000 UTC m=+1377.994828164" Sep 08 18:57:30.085064 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.085024 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk"] Sep 08 18:57:30.090654 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.090633 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" Sep 08 18:57:30.093174 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.093147 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-525196-dockercfg-rr2vz\"" Sep 08 18:57:30.093284 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.093183 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-525196\"" Sep 08 18:57:30.097400 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.097356 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk"] Sep 08 18:57:30.106860 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.106836 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7"] Sep 08 18:57:30.110032 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.110011 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" Sep 08 18:57:30.121497 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.121472 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7"] Sep 08 18:57:30.203618 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.203567 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a282b530-9233-4a7c-9104-020177cef1aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7\" (UID: \"a282b530-9233-4a7c-9104-020177cef1aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" Sep 08 18:57:30.203808 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.203644 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde2aafc-0c41-47d1-b928-9541ecb0dd17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk\" (UID: \"dde2aafc-0c41-47d1-b928-9541ecb0dd17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" Sep 08 18:57:30.203808 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.203760 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfqpv\" (UniqueName: \"kubernetes.io/projected/dde2aafc-0c41-47d1-b928-9541ecb0dd17-kube-api-access-zfqpv\") pod \"payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk\" (UID: \"dde2aafc-0c41-47d1-b928-9541ecb0dd17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" Sep 08 18:57:30.203917 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.203835 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxzgv\" (UniqueName: \"kubernetes.io/projected/a282b530-9233-4a7c-9104-020177cef1aa-kube-api-access-bxzgv\") pod \"payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7\" (UID: \"a282b530-9233-4a7c-9104-020177cef1aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" Sep 08 18:57:30.305233 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.305194 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfqpv\" (UniqueName: \"kubernetes.io/projected/dde2aafc-0c41-47d1-b928-9541ecb0dd17-kube-api-access-zfqpv\") pod \"payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk\" (UID: \"dde2aafc-0c41-47d1-b928-9541ecb0dd17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" Sep 08 18:57:30.305458 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.305240 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxzgv\" (UniqueName: \"kubernetes.io/projected/a282b530-9233-4a7c-9104-020177cef1aa-kube-api-access-bxzgv\") pod \"payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7\" (UID: \"a282b530-9233-4a7c-9104-020177cef1aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" Sep 08 18:57:30.305458 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.305286 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a282b530-9233-4a7c-9104-020177cef1aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7\" (UID: \"a282b530-9233-4a7c-9104-020177cef1aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" Sep 08 18:57:30.305458 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.305304 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde2aafc-0c41-47d1-b928-9541ecb0dd17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk\" (UID: \"dde2aafc-0c41-47d1-b928-9541ecb0dd17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" Sep 08 18:57:30.306229 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.306198 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a282b530-9233-4a7c-9104-020177cef1aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7\" (UID: \"a282b530-9233-4a7c-9104-020177cef1aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" Sep 08 18:57:30.306323 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.306198 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde2aafc-0c41-47d1-b928-9541ecb0dd17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk\" (UID: \"dde2aafc-0c41-47d1-b928-9541ecb0dd17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" Sep 08 18:57:30.314488 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.314459 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfqpv\" (UniqueName: \"kubernetes.io/projected/dde2aafc-0c41-47d1-b928-9541ecb0dd17-kube-api-access-zfqpv\") pod \"payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk\" (UID: \"dde2aafc-0c41-47d1-b928-9541ecb0dd17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" Sep 08 18:57:30.314933 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.314915 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxzgv\" (UniqueName: \"kubernetes.io/projected/a282b530-9233-4a7c-9104-020177cef1aa-kube-api-access-bxzgv\") pod \"payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7\" (UID: \"a282b530-9233-4a7c-9104-020177cef1aa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" Sep 08 18:57:30.401760 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.401729 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" Sep 08 18:57:30.422059 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.422031 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" Sep 08 18:57:30.676629 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.676599 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk"] Sep 08 18:57:30.677686 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:57:30.677663 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde2aafc_0c41_47d1_b928_9541ecb0dd17.slice/crio-6b18673f8b82cd6f5886ed3da71ed997b434ab4ed7efddeea470a35be8722bfa WatchSource:0}: Error finding container 6b18673f8b82cd6f5886ed3da71ed997b434ab4ed7efddeea470a35be8722bfa: Status 404 returned error can't find the container with id 6b18673f8b82cd6f5886ed3da71ed997b434ab4ed7efddeea470a35be8722bfa Sep 08 18:57:30.709770 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:30.709744 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7"] Sep 08 18:57:30.710177 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:57:30.710156 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda282b530_9233_4a7c_9104_020177cef1aa.slice/crio-ccc742d34f9f1608d0d07cc2f310864ee297d1462532b85f90b0a6b18a359d4e WatchSource:0}: Error finding container ccc742d34f9f1608d0d07cc2f310864ee297d1462532b85f90b0a6b18a359d4e: Status 404 returned error can't find the container with id ccc742d34f9f1608d0d07cc2f310864ee297d1462532b85f90b0a6b18a359d4e Sep 08 18:57:31.320607 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:31.320564 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" event={"ID":"a282b530-9233-4a7c-9104-020177cef1aa","Type":"ContainerStarted","Data":"eeda73dd3d94df67dbca5f3628bfb8ec7b6f18475d4226986c35a59203b7966f"} Sep 08 18:57:31.321045 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:31.320636 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" Sep 08 18:57:31.321045 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:31.320660 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" event={"ID":"a282b530-9233-4a7c-9104-020177cef1aa","Type":"ContainerStarted","Data":"ccc742d34f9f1608d0d07cc2f310864ee297d1462532b85f90b0a6b18a359d4e"} Sep 08 18:57:31.322314 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:31.322291 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" event={"ID":"dde2aafc-0c41-47d1-b928-9541ecb0dd17","Type":"ContainerStarted","Data":"06ce1b098daf87a1e7df736f6b63d25a219d5a811f2549a9cd08a4781247da1f"} Sep 08 18:57:31.322438 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:31.322334 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" event={"ID":"dde2aafc-0c41-47d1-b928-9541ecb0dd17","Type":"ContainerStarted","Data":"6b18673f8b82cd6f5886ed3da71ed997b434ab4ed7efddeea470a35be8722bfa"} Sep 08 18:57:31.322438 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:31.322395 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" Sep 08 18:57:31.339598 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:31.339545 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" podStartSLOduration=1.339527451 podStartE2EDuration="1.339527451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:31.33679079 +0000 UTC m=+1387.042679024" watchObservedRunningTime="2026-09-08 18:57:31.339527451 +0000 UTC m=+1387.045415685" Sep 08 18:57:31.354696 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:31.354652 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" podStartSLOduration=1.3546380120000001 podStartE2EDuration="1.354638012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:31.352393426 +0000 UTC m=+1387.058281661" watchObservedRunningTime="2026-09-08 18:57:31.354638012 +0000 UTC m=+1387.060526243" Sep 08 18:57:33.276721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:33.276693 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" Sep 08 18:57:42.330890 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:42.330854 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" Sep 08 18:57:42.331253 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:42.330934 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" Sep 08 18:57:44.914010 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:44.913979 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8"] Sep 08 18:57:44.917998 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:44.917976 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:44.920738 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:44.920717 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c6f2ed-openshift-default-dockercfg-z82l5\"" Sep 08 18:57:44.928102 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:44.928073 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8"] Sep 08 18:57:45.034062 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.034026 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-workload-certs\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.034263 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.034075 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-envoy\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.034263 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.034107 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-data\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.034263 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.034141 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwwtv\" (UniqueName: \"kubernetes.io/projected/655bc981-3bba-4de0-ac3b-513b13b896ad-kube-api-access-rwwtv\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.034263 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.034207 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-podinfo\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.034263 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.034257 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655bc981-3bba-4de0-ac3b-513b13b896ad-istiod-ca-cert\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.034847 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.034283 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-workload-socket\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.034847 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.034312 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-credential-socket\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.034847 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.034428 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-token\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.135073 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.135039 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-workload-certs\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.135073 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.135077 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-envoy\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.135285 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.135102 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-data\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.135285 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.135144 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwwtv\" (UniqueName: \"kubernetes.io/projected/655bc981-3bba-4de0-ac3b-513b13b896ad-kube-api-access-rwwtv\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.135285 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.135182 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-podinfo\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.135285 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.135209 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655bc981-3bba-4de0-ac3b-513b13b896ad-istiod-ca-cert\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.135516 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.135396 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-workload-socket\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.135516 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.135448 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-credential-socket\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.135621 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.135530 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-workload-certs\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.135621 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.135544 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-token\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.135721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.135619 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-data\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.135721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.135664 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-workload-socket\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.135823 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.135800 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-credential-socket\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.135911 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.135894 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655bc981-3bba-4de0-ac3b-513b13b896ad-istiod-ca-cert\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.137599 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.137569 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-envoy\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.137713 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.137632 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-podinfo\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.142626 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.142599 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-token\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.142807 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.142790 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwwtv\" (UniqueName: \"kubernetes.io/projected/655bc981-3bba-4de0-ac3b-513b13b896ad-kube-api-access-rwwtv\") pod \"e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.229921 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.229844 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:45.513154 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.513122 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8"] Sep 08 18:57:45.514294 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:57:45.514260 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655bc981_3bba_4de0_ac3b_513b13b896ad.slice/crio-e390257c9ee9c8c905802faf08347289a744ccd5bd6a439fa0e8a79d3caf60a1 WatchSource:0}: Error finding container e390257c9ee9c8c905802faf08347289a744ccd5bd6a439fa0e8a79d3caf60a1: Status 404 returned error can't find the container with id e390257c9ee9c8c905802faf08347289a744ccd5bd6a439fa0e8a79d3caf60a1 Sep 08 18:57:45.519665 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.519632 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:57:45.519757 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.519697 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:57:45.519757 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:45.519725 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:57:46.414355 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:46.414320 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" event={"ID":"655bc981-3bba-4de0-ac3b-513b13b896ad","Type":"ContainerStarted","Data":"d74d9c437ee799093b58a7505ba03ee96ef3f8a82fbdafe06f7420de73e2c42f"} Sep 08 18:57:46.414850 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:46.414391 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" event={"ID":"655bc981-3bba-4de0-ac3b-513b13b896ad","Type":"ContainerStarted","Data":"e390257c9ee9c8c905802faf08347289a744ccd5bd6a439fa0e8a79d3caf60a1"} Sep 08 18:57:46.440594 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:46.440534 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" podStartSLOduration=2.44050843 podStartE2EDuration="2.44050843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:46.435705644 +0000 UTC m=+1402.141593875" watchObservedRunningTime="2026-09-08 18:57:46.44050843 +0000 UTC m=+1402.146396668" Sep 08 18:57:47.230693 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:47.230656 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:47.235495 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:47.235468 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:47.420677 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:47.420641 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:47.421657 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:47.421638 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:57:50.240200 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:50.240097 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp"] Sep 08 18:57:50.243810 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:50.243787 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" Sep 08 18:57:50.246586 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:50.246563 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c6f2ed\"" Sep 08 18:57:50.246719 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:50.246596 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c6f2ed-dockercfg-j66pk\"" Sep 08 18:57:50.265611 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:50.265584 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp"] Sep 08 18:57:50.383003 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:50.382970 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfd8p\" (UniqueName: \"kubernetes.io/projected/4b317876-c273-434c-8ff1-9f3b6637a72d-kube-api-access-cfd8p\") pod \"payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp\" (UID: \"4b317876-c273-434c-8ff1-9f3b6637a72d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" Sep 08 18:57:50.383188 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:50.383022 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b317876-c273-434c-8ff1-9f3b6637a72d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp\" (UID: \"4b317876-c273-434c-8ff1-9f3b6637a72d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" Sep 08 18:57:50.484478 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:50.484440 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfd8p\" (UniqueName: \"kubernetes.io/projected/4b317876-c273-434c-8ff1-9f3b6637a72d-kube-api-access-cfd8p\") pod \"payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp\" (UID: \"4b317876-c273-434c-8ff1-9f3b6637a72d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" Sep 08 18:57:50.484670 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:50.484502 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b317876-c273-434c-8ff1-9f3b6637a72d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp\" (UID: \"4b317876-c273-434c-8ff1-9f3b6637a72d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" Sep 08 18:57:50.485176 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:50.485157 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b317876-c273-434c-8ff1-9f3b6637a72d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp\" (UID: \"4b317876-c273-434c-8ff1-9f3b6637a72d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" Sep 08 18:57:50.493935 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:50.493882 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfd8p\" (UniqueName: \"kubernetes.io/projected/4b317876-c273-434c-8ff1-9f3b6637a72d-kube-api-access-cfd8p\") pod \"payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp\" (UID: \"4b317876-c273-434c-8ff1-9f3b6637a72d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" Sep 08 18:57:50.555422 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:50.555397 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" Sep 08 18:57:50.829320 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:50.829290 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp"] Sep 08 18:57:50.830278 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:57:50.830252 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b317876_c273_434c_8ff1_9f3b6637a72d.slice/crio-a50ffd45bd8c1d314e2a5d3df96614de7c1e1c5a6dabede2eb93813a8e5de22f WatchSource:0}: Error finding container a50ffd45bd8c1d314e2a5d3df96614de7c1e1c5a6dabede2eb93813a8e5de22f: Status 404 returned error can't find the container with id a50ffd45bd8c1d314e2a5d3df96614de7c1e1c5a6dabede2eb93813a8e5de22f Sep 08 18:57:51.446960 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:51.446920 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" event={"ID":"4b317876-c273-434c-8ff1-9f3b6637a72d","Type":"ContainerStarted","Data":"4190f0450f49a38030b54610edbde817330e527e50f74ee36033c87ef798926e"} Sep 08 18:57:51.447426 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:51.446989 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" event={"ID":"4b317876-c273-434c-8ff1-9f3b6637a72d","Type":"ContainerStarted","Data":"a50ffd45bd8c1d314e2a5d3df96614de7c1e1c5a6dabede2eb93813a8e5de22f"} Sep 08 18:57:51.447426 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:51.447112 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" Sep 08 18:57:51.469581 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:57:51.469529 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" podStartSLOduration=1.469513175 podStartE2EDuration="1.469513175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 18:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 18:57:51.466045064 +0000 UTC m=+1407.171933296" watchObservedRunningTime="2026-09-08 18:57:51.469513175 +0000 UTC m=+1407.175401437" Sep 08 18:58:02.455296 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:02.455262 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" Sep 08 18:58:06.539907 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:06.539821 2553 generic.go:356] "Generic (PLEG): container finished" podID="655bc981-3bba-4de0-ac3b-513b13b896ad" containerID="d74d9c437ee799093b58a7505ba03ee96ef3f8a82fbdafe06f7420de73e2c42f" exitCode=0 Sep 08 18:58:06.540440 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:06.539897 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" event={"ID":"655bc981-3bba-4de0-ac3b-513b13b896ad","Type":"ContainerDied","Data":"d74d9c437ee799093b58a7505ba03ee96ef3f8a82fbdafe06f7420de73e2c42f"} Sep 08 18:58:06.540440 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:06.540413 2553 scope.go:117] "RemoveContainer" containerID="d74d9c437ee799093b58a7505ba03ee96ef3f8a82fbdafe06f7420de73e2c42f" Sep 08 18:58:06.541528 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:06.541497 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:58:06.541620 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:06.541565 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:58:06.541620 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:06.541596 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 18:58:07.553855 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:07.553815 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" event={"ID":"655bc981-3bba-4de0-ac3b-513b13b896ad","Type":"ContainerStarted","Data":"d4771622673c93f36419b1ac3a6b6e2a240d89c0c87b999fa0d03de4d381c0f0"} Sep 08 18:58:08.230518 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:08.230481 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:58:08.230884 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:08.230858 2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 08 18:58:08.231081 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:08.231053 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" podUID="655bc981-3bba-4de0-ac3b-513b13b896ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 08 18:58:09.231143 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:09.231098 2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 08 18:58:09.231658 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:09.231179 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" podUID="655bc981-3bba-4de0-ac3b-513b13b896ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 08 18:58:10.231780 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:10.231304 2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 08 18:58:10.231780 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:10.231398 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" podUID="655bc981-3bba-4de0-ac3b-513b13b896ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 08 18:58:11.230795 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:11.230751 2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 08 18:58:11.231129 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:11.231099 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" podUID="655bc981-3bba-4de0-ac3b-513b13b896ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 08 18:58:12.230449 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:12.230406 2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 08 18:58:12.230966 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:12.230477 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" podUID="655bc981-3bba-4de0-ac3b-513b13b896ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 08 18:58:13.234923 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:13.234886 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:58:13.235294 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:13.235267 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:58:13.236053 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:58:13.236029 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 18:59:24.919422 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:24.919393 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 18:59:24.922566 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:24.922541 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 18:59:24.923425 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:24.923402 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 18:59:24.926220 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:24.926199 2553 scope.go:117] "RemoveContainer" containerID="4d6b0e63b2cbd671f1b7290911196f33694d442f1584db6cd43b7ef83fcc40ad" Sep 08 18:59:24.926634 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:24.926616 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 18:59:24.940445 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:24.940422 2553 scope.go:117] "RemoveContainer" containerID="b7624a953a815cc6b3f5ac5c761c5da408f5ccefba1f014edd931e1aebb0574d" Sep 08 18:59:46.679810 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:46.679768 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3adca991-lxp88"] Sep 08 18:59:46.683025 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:46.683003 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3adca991-lxp88" Sep 08 18:59:46.686171 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:46.686150 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 18:59:46.686613 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:46.686169 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 18:59:46.688721 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:46.688701 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3adca991-lxp88"] Sep 08 18:59:46.739204 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:46.739165 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f5f4a4-c553-4c7a-b02d-6b3fa5365d63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3adca991-lxp88\" (UID: \"50f5f4a4-c553-4c7a-b02d-6b3fa5365d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3adca991-lxp88" Sep 08 18:59:46.840021 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:46.839983 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f5f4a4-c553-4c7a-b02d-6b3fa5365d63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3adca991-lxp88\" (UID: \"50f5f4a4-c553-4c7a-b02d-6b3fa5365d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3adca991-lxp88" Sep 08 18:59:46.840676 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:46.840649 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f5f4a4-c553-4c7a-b02d-6b3fa5365d63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3adca991-lxp88\" (UID: \"50f5f4a4-c553-4c7a-b02d-6b3fa5365d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3adca991-lxp88" Sep 08 18:59:46.997161 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:46.997080 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3adca991-lxp88" Sep 08 18:59:47.284311 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:47.284271 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3adca991-lxp88"] Sep 08 18:59:47.285482 ip-10-0-132-43 kubenswrapper[2553]: W0908 18:59:47.285456 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f5f4a4_c553_4c7a_b02d_6b3fa5365d63.slice/crio-bab900dc69e7eeda6bf7cca8ece600799ae97caacda2b4efec6f8a5b3288f594 WatchSource:0}: Error finding container bab900dc69e7eeda6bf7cca8ece600799ae97caacda2b4efec6f8a5b3288f594: Status 404 returned error can't find the container with id bab900dc69e7eeda6bf7cca8ece600799ae97caacda2b4efec6f8a5b3288f594 Sep 08 18:59:48.126477 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:48.126440 2553 generic.go:356] "Generic (PLEG): container finished" podID="50f5f4a4-c553-4c7a-b02d-6b3fa5365d63" containerID="00c3dc8c58ae3e9bc2f82dac9d612072f3dcb207b8ba79d82532410ca8318dcf" exitCode=0 Sep 08 18:59:48.126817 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:48.126480 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3adca991-lxp88" event={"ID":"50f5f4a4-c553-4c7a-b02d-6b3fa5365d63","Type":"ContainerDied","Data":"00c3dc8c58ae3e9bc2f82dac9d612072f3dcb207b8ba79d82532410ca8318dcf"} Sep 08 18:59:48.126817 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:48.126533 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3adca991-lxp88" event={"ID":"50f5f4a4-c553-4c7a-b02d-6b3fa5365d63","Type":"ContainerStarted","Data":"bab900dc69e7eeda6bf7cca8ece600799ae97caacda2b4efec6f8a5b3288f594"} Sep 08 18:59:49.269202 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:49.269178 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3adca991-lxp88" Sep 08 18:59:49.365818 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:49.365784 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f5f4a4-c553-4c7a-b02d-6b3fa5365d63-maas-api-service-ca\") pod \"50f5f4a4-c553-4c7a-b02d-6b3fa5365d63\" (UID: \"50f5f4a4-c553-4c7a-b02d-6b3fa5365d63\") " Sep 08 18:59:49.366095 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:49.366072 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f5f4a4-c553-4c7a-b02d-6b3fa5365d63-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50f5f4a4-c553-4c7a-b02d-6b3fa5365d63" (UID: "50f5f4a4-c553-4c7a-b02d-6b3fa5365d63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 18:59:49.466754 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:49.466672 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f5f4a4-c553-4c7a-b02d-6b3fa5365d63-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 18:59:50.138777 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:50.138741 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3adca991-lxp88" event={"ID":"50f5f4a4-c553-4c7a-b02d-6b3fa5365d63","Type":"ContainerDied","Data":"bab900dc69e7eeda6bf7cca8ece600799ae97caacda2b4efec6f8a5b3288f594"} Sep 08 18:59:50.138777 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:50.138768 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3adca991-lxp88" Sep 08 18:59:50.138980 ip-10-0-132-43 kubenswrapper[2553]: I0908 18:59:50.138791 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bab900dc69e7eeda6bf7cca8ece600799ae97caacda2b4efec6f8a5b3288f594" Sep 08 19:00:00.142153 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.142116 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4001b-29814900-d6z5c"] Sep 08 19:00:00.142771 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.142749 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f5f4a4-c553-4c7a-b02d-6b3fa5365d63" containerName="revoke-keys" Sep 08 19:00:00.142862 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.142773 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f5f4a4-c553-4c7a-b02d-6b3fa5365d63" containerName="revoke-keys" Sep 08 19:00:00.142921 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.142886 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f5f4a4-c553-4c7a-b02d-6b3fa5365d63" containerName="revoke-keys" Sep 08 19:00:00.144989 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.144968 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c6f2ed-29814900-n4vsx"] Sep 08 19:00:00.145128 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.145106 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4001b-29814900-d6z5c" Sep 08 19:00:00.147979 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.147959 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 19:00:00.149207 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.149190 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-525196-29814900-fgrmv"] Sep 08 19:00:00.149331 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.149313 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c6f2ed-29814900-n4vsx" Sep 08 19:00:00.156327 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.155626 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814900-nbcwm"] Sep 08 19:00:00.156872 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.156853 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-525196-29814900-fgrmv" Sep 08 19:00:00.159065 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.158738 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4001b-29814900-d6z5c"] Sep 08 19:00:00.159175 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.159118 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-525196-29814900-fgrmv"] Sep 08 19:00:00.159175 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.158805 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-nbcwm" Sep 08 19:00:00.162388 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.162347 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c6f2ed-29814900-n4vsx"] Sep 08 19:00:00.162701 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.162676 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4001b-29814900-d6z5c" Sep 08 19:00:00.180905 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.180877 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c6f2ed-29814900-n4vsx" Sep 08 19:00:00.182327 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.182306 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814900-nbcwm"] Sep 08 19:00:00.218009 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.217969 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-525196-29814900-fgrmv" Sep 08 19:00:00.224754 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.224626 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-nbcwm" Sep 08 19:00:00.462969 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.462931 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4001b-29814900-d6z5c"] Sep 08 19:00:00.464874 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:00:00.464836 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22456267_acb6_466a_a1b8_e4123fb861ac.slice/crio-534f79bea04e55d33a24bb6ae688b3570a5e982087dd49414965990a2b172d16 WatchSource:0}: Error finding container 534f79bea04e55d33a24bb6ae688b3570a5e982087dd49414965990a2b172d16: Status 404 returned error can't find the container with id 534f79bea04e55d33a24bb6ae688b3570a5e982087dd49414965990a2b172d16 Sep 08 19:00:00.477843 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.477819 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c6f2ed-29814900-n4vsx"] Sep 08 19:00:00.481708 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:00:00.481679 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3d6801_46b4_42fb_9887_eb8202523e54.slice/crio-1d72aa566f976fcd6cda6fffbaa197cf18b6224a08f223dc9cea00bac27ff42c WatchSource:0}: Error finding container 1d72aa566f976fcd6cda6fffbaa197cf18b6224a08f223dc9cea00bac27ff42c: Status 404 returned error can't find the container with id 1d72aa566f976fcd6cda6fffbaa197cf18b6224a08f223dc9cea00bac27ff42c Sep 08 19:00:00.515287 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.515260 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-525196-29814900-fgrmv"] Sep 08 19:00:00.516869 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:00:00.516840 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded71a480_4e6b_4247_9dad_4c66e270066f.slice/crio-77e000cb397540f09d9b6b604fa4025ac1e5389b37562bc22ec117e6bda1fb11 WatchSource:0}: Error finding container 77e000cb397540f09d9b6b604fa4025ac1e5389b37562bc22ec117e6bda1fb11: Status 404 returned error can't find the container with id 77e000cb397540f09d9b6b604fa4025ac1e5389b37562bc22ec117e6bda1fb11 Sep 08 19:00:00.524399 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:00.520482 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814900-nbcwm"] Sep 08 19:00:01.208043 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:01.208003 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4001b-29814900-d6z5c" event={"ID":"22456267-acb6-466a-a1b8-e4123fb861ac","Type":"ContainerStarted","Data":"d867dba17242da94e291752c08a45b964f19eda72188e56e24ca149e8c783c08"} Sep 08 19:00:01.208431 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:01.208067 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4001b-29814900-d6z5c" event={"ID":"22456267-acb6-466a-a1b8-e4123fb861ac","Type":"ContainerStarted","Data":"534f79bea04e55d33a24bb6ae688b3570a5e982087dd49414965990a2b172d16"} Sep 08 19:00:01.209712 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:01.209685 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c6f2ed-29814900-n4vsx" event={"ID":"8b3d6801-46b4-42fb-9887-eb8202523e54","Type":"ContainerStarted","Data":"d7035e3747c0acd8fbcff0065e454feb59d0146853a4182decc4ab10c6ca3fa4"} Sep 08 19:00:01.209838 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:01.209737 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c6f2ed-29814900-n4vsx" event={"ID":"8b3d6801-46b4-42fb-9887-eb8202523e54","Type":"ContainerStarted","Data":"1d72aa566f976fcd6cda6fffbaa197cf18b6224a08f223dc9cea00bac27ff42c"} Sep 08 19:00:01.211358 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:01.211330 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-nbcwm" event={"ID":"dff884b1-aad4-4b77-9fbb-5e1d02b41f74","Type":"ContainerStarted","Data":"f4b29bb9cb6532e6c4e0bc7d71840339326fbf6a218562ed34780c5d8f16bb1b"} Sep 08 19:00:01.211507 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:01.211392 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-nbcwm" event={"ID":"dff884b1-aad4-4b77-9fbb-5e1d02b41f74","Type":"ContainerStarted","Data":"5224675765f7dccb948044ed6aaac25b1439763e2e145f9bba6c7f7e4a9c76bf"} Sep 08 19:00:01.212777 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:01.212752 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-525196-29814900-fgrmv" event={"ID":"ed71a480-4e6b-4247-9dad-4c66e270066f","Type":"ContainerStarted","Data":"2e7104f97d3bfea34482865a2a473c8442af15974d962d28ec7f1945fb269872"} Sep 08 19:00:01.212880 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:01.212797 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-525196-29814900-fgrmv" event={"ID":"ed71a480-4e6b-4247-9dad-4c66e270066f","Type":"ContainerStarted","Data":"77e000cb397540f09d9b6b604fa4025ac1e5389b37562bc22ec117e6bda1fb11"} Sep 08 19:00:01.225607 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:01.225559 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4001b-29814900-d6z5c" podStartSLOduration=1.225546787 podStartE2EDuration="1.225546787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:00:01.22210596 +0000 UTC m=+1536.927994249" watchObservedRunningTime="2026-09-08 19:00:01.225546787 +0000 UTC m=+1536.931435047" Sep 08 19:00:01.238867 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:01.238819 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c6f2ed-29814900-n4vsx" podStartSLOduration=1.23880598 podStartE2EDuration="1.23880598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:00:01.236708046 +0000 UTC m=+1536.942596280" watchObservedRunningTime="2026-09-08 19:00:01.23880598 +0000 UTC m=+1536.944694212" Sep 08 19:00:01.251760 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:01.251719 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-525196-29814900-fgrmv" podStartSLOduration=1.251706118 podStartE2EDuration="1.251706118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:00:01.250432621 +0000 UTC m=+1536.956320854" watchObservedRunningTime="2026-09-08 19:00:01.251706118 +0000 UTC m=+1536.957594388" Sep 08 19:00:01.269435 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:01.269388 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-nbcwm" podStartSLOduration=1.269378823 podStartE2EDuration="1.269378823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:00:01.266620053 +0000 UTC m=+1536.972508290" watchObservedRunningTime="2026-09-08 19:00:01.269378823 +0000 UTC m=+1536.975267046" Sep 08 19:00:02.220775 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:02.220736 2553 generic.go:356] "Generic (PLEG): container finished" podID="8b3d6801-46b4-42fb-9887-eb8202523e54" containerID="d7035e3747c0acd8fbcff0065e454feb59d0146853a4182decc4ab10c6ca3fa4" exitCode=0 Sep 08 19:00:02.221244 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:02.220812 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c6f2ed-29814900-n4vsx" event={"ID":"8b3d6801-46b4-42fb-9887-eb8202523e54","Type":"ContainerDied","Data":"d7035e3747c0acd8fbcff0065e454feb59d0146853a4182decc4ab10c6ca3fa4"} Sep 08 19:00:02.223230 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:02.223203 2553 generic.go:356] "Generic (PLEG): container finished" podID="dff884b1-aad4-4b77-9fbb-5e1d02b41f74" containerID="f4b29bb9cb6532e6c4e0bc7d71840339326fbf6a218562ed34780c5d8f16bb1b" exitCode=0 Sep 08 19:00:02.223385 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:02.223256 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-nbcwm" event={"ID":"dff884b1-aad4-4b77-9fbb-5e1d02b41f74","Type":"ContainerDied","Data":"f4b29bb9cb6532e6c4e0bc7d71840339326fbf6a218562ed34780c5d8f16bb1b"} Sep 08 19:00:02.225228 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:02.225202 2553 generic.go:356] "Generic (PLEG): container finished" podID="ed71a480-4e6b-4247-9dad-4c66e270066f" containerID="2e7104f97d3bfea34482865a2a473c8442af15974d962d28ec7f1945fb269872" exitCode=0 Sep 08 19:00:02.225337 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:02.225304 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-525196-29814900-fgrmv" event={"ID":"ed71a480-4e6b-4247-9dad-4c66e270066f","Type":"ContainerDied","Data":"2e7104f97d3bfea34482865a2a473c8442af15974d962d28ec7f1945fb269872"} Sep 08 19:00:02.227768 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:02.227743 2553 generic.go:356] "Generic (PLEG): container finished" podID="22456267-acb6-466a-a1b8-e4123fb861ac" containerID="d867dba17242da94e291752c08a45b964f19eda72188e56e24ca149e8c783c08" exitCode=0 Sep 08 19:00:02.227918 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:02.227899 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4001b-29814900-d6z5c" event={"ID":"22456267-acb6-466a-a1b8-e4123fb861ac","Type":"ContainerDied","Data":"d867dba17242da94e291752c08a45b964f19eda72188e56e24ca149e8c783c08"} Sep 08 19:00:02.379616 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:02.379579 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3adca991-lxp88"] Sep 08 19:00:02.384063 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:02.384036 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3adca991-lxp88"] Sep 08 19:00:02.746064 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:02.746030 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f5f4a4-c553-4c7a-b02d-6b3fa5365d63" path="/var/lib/kubelet/pods/50f5f4a4-c553-4c7a-b02d-6b3fa5365d63/volumes" Sep 08 19:00:03.399481 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:03.399436 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-nbcwm" Sep 08 19:00:03.457423 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:03.457404 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c6f2ed-29814900-n4vsx" Sep 08 19:00:03.460799 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:03.460772 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4001b-29814900-d6z5c" Sep 08 19:00:03.463899 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:03.463882 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-525196-29814900-fgrmv" Sep 08 19:00:04.247390 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:04.247343 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c6f2ed-29814900-n4vsx" Sep 08 19:00:04.247562 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:04.247390 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c6f2ed-29814900-n4vsx" event={"ID":"8b3d6801-46b4-42fb-9887-eb8202523e54","Type":"ContainerDied","Data":"1d72aa566f976fcd6cda6fffbaa197cf18b6224a08f223dc9cea00bac27ff42c"} Sep 08 19:00:04.247562 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:04.247460 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d72aa566f976fcd6cda6fffbaa197cf18b6224a08f223dc9cea00bac27ff42c" Sep 08 19:00:04.248859 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:04.248836 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-nbcwm" Sep 08 19:00:04.248998 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:04.248833 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814900-nbcwm" event={"ID":"dff884b1-aad4-4b77-9fbb-5e1d02b41f74","Type":"ContainerDied","Data":"5224675765f7dccb948044ed6aaac25b1439763e2e145f9bba6c7f7e4a9c76bf"} Sep 08 19:00:04.248998 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:04.248951 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5224675765f7dccb948044ed6aaac25b1439763e2e145f9bba6c7f7e4a9c76bf" Sep 08 19:00:04.250823 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:04.250722 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-525196-29814900-fgrmv" Sep 08 19:00:04.250823 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:04.250742 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-525196-29814900-fgrmv" event={"ID":"ed71a480-4e6b-4247-9dad-4c66e270066f","Type":"ContainerDied","Data":"77e000cb397540f09d9b6b604fa4025ac1e5389b37562bc22ec117e6bda1fb11"} Sep 08 19:00:04.250823 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:04.250786 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77e000cb397540f09d9b6b604fa4025ac1e5389b37562bc22ec117e6bda1fb11" Sep 08 19:00:04.252565 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:04.252538 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4001b-29814900-d6z5c" event={"ID":"22456267-acb6-466a-a1b8-e4123fb861ac","Type":"ContainerDied","Data":"534f79bea04e55d33a24bb6ae688b3570a5e982087dd49414965990a2b172d16"} Sep 08 19:00:04.252655 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:04.252591 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="534f79bea04e55d33a24bb6ae688b3570a5e982087dd49414965990a2b172d16" Sep 08 19:00:04.252655 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:04.252616 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4001b-29814900-d6z5c" Sep 08 19:00:24.965190 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:24.965157 2553 scope.go:117] "RemoveContainer" containerID="7bae01e7a5b92e602d46630c610a1ff8cce512d2dd225b2f5608a73a8d1a4d46" Sep 08 19:00:24.974069 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:24.974048 2553 scope.go:117] "RemoveContainer" containerID="a9d9123ce650e22608da88b24874f0095bfcc32a8b5277b7e2f4b3d87d218af6" Sep 08 19:00:24.982628 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:00:24.982603 2553 scope.go:117] "RemoveContainer" containerID="230b0b1976f05308948afd6cea5c89303a232f7542245c24ba28826eabb60d9e" Sep 08 19:01:25.014553 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:01:25.014474 2553 scope.go:117] "RemoveContainer" containerID="83fcbdeceb13c145674fdfa4fe61b3107f2e7d2a1d60ed26981decc3f5822b87" Sep 08 19:02:08.873140 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.873101 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw"] Sep 08 19:02:08.873611 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.873504 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b3d6801-46b4-42fb-9887-eb8202523e54" containerName="cleanup" Sep 08 19:02:08.873611 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.873516 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b3d6801-46b4-42fb-9887-eb8202523e54" containerName="cleanup" Sep 08 19:02:08.873611 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.873533 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed71a480-4e6b-4247-9dad-4c66e270066f" containerName="cleanup" Sep 08 19:02:08.873611 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.873538 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed71a480-4e6b-4247-9dad-4c66e270066f" containerName="cleanup" Sep 08 19:02:08.873611 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.873549 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22456267-acb6-466a-a1b8-e4123fb861ac" containerName="cleanup" Sep 08 19:02:08.873611 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.873555 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="22456267-acb6-466a-a1b8-e4123fb861ac" containerName="cleanup" Sep 08 19:02:08.873611 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.873564 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff884b1-aad4-4b77-9fbb-5e1d02b41f74" containerName="cleanup" Sep 08 19:02:08.873611 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.873569 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff884b1-aad4-4b77-9fbb-5e1d02b41f74" containerName="cleanup" Sep 08 19:02:08.873865 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.873629 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed71a480-4e6b-4247-9dad-4c66e270066f" containerName="cleanup" Sep 08 19:02:08.873865 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.873636 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="22456267-acb6-466a-a1b8-e4123fb861ac" containerName="cleanup" Sep 08 19:02:08.873865 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.873644 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b3d6801-46b4-42fb-9887-eb8202523e54" containerName="cleanup" Sep 08 19:02:08.873865 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.873650 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff884b1-aad4-4b77-9fbb-5e1d02b41f74" containerName="cleanup" Sep 08 19:02:08.876689 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.876673 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" Sep 08 19:02:08.879508 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.879487 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-965746\"" Sep 08 19:02:08.879612 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.879555 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-965746-dockercfg-cnb68\"" Sep 08 19:02:08.890689 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.890665 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw"] Sep 08 19:02:08.941955 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.941921 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bfa122d-8806-486c-b2ed-a682095514c1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-965746-dc45866d5-j5wlw\" (UID: \"5bfa122d-8806-486c-b2ed-a682095514c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" Sep 08 19:02:08.942096 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:08.941958 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrc7k\" (UniqueName: \"kubernetes.io/projected/5bfa122d-8806-486c-b2ed-a682095514c1-kube-api-access-xrc7k\") pod \"payload-processing-e2e-api-a-965746-dc45866d5-j5wlw\" (UID: \"5bfa122d-8806-486c-b2ed-a682095514c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" Sep 08 19:02:09.043192 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:09.043154 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bfa122d-8806-486c-b2ed-a682095514c1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-965746-dc45866d5-j5wlw\" (UID: \"5bfa122d-8806-486c-b2ed-a682095514c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" Sep 08 19:02:09.043192 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:09.043189 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrc7k\" (UniqueName: \"kubernetes.io/projected/5bfa122d-8806-486c-b2ed-a682095514c1-kube-api-access-xrc7k\") pod \"payload-processing-e2e-api-a-965746-dc45866d5-j5wlw\" (UID: \"5bfa122d-8806-486c-b2ed-a682095514c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" Sep 08 19:02:09.043907 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:09.043885 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bfa122d-8806-486c-b2ed-a682095514c1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-965746-dc45866d5-j5wlw\" (UID: \"5bfa122d-8806-486c-b2ed-a682095514c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" Sep 08 19:02:09.052023 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:09.051998 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrc7k\" (UniqueName: \"kubernetes.io/projected/5bfa122d-8806-486c-b2ed-a682095514c1-kube-api-access-xrc7k\") pod \"payload-processing-e2e-api-a-965746-dc45866d5-j5wlw\" (UID: \"5bfa122d-8806-486c-b2ed-a682095514c1\") " pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" Sep 08 19:02:09.186903 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:09.186831 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" Sep 08 19:02:09.463850 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:09.463814 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw"] Sep 08 19:02:09.464175 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:02:09.464154 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfa122d_8806_486c_b2ed_a682095514c1.slice/crio-c88cb2d9a00e9b66d369d0e52cbce8e48f93b3d33d42a171368cbdfd459c4b93 WatchSource:0}: Error finding container c88cb2d9a00e9b66d369d0e52cbce8e48f93b3d33d42a171368cbdfd459c4b93: Status 404 returned error can't find the container with id c88cb2d9a00e9b66d369d0e52cbce8e48f93b3d33d42a171368cbdfd459c4b93 Sep 08 19:02:10.018496 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:10.018461 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" event={"ID":"5bfa122d-8806-486c-b2ed-a682095514c1","Type":"ContainerStarted","Data":"be61b3dc184002922aedf42551087717b2746b46827bdbdc5b2b5f9b8bfaaf09"} Sep 08 19:02:10.018963 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:10.018514 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" event={"ID":"5bfa122d-8806-486c-b2ed-a682095514c1","Type":"ContainerStarted","Data":"c88cb2d9a00e9b66d369d0e52cbce8e48f93b3d33d42a171368cbdfd459c4b93"} Sep 08 19:02:10.018963 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:10.018549 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" Sep 08 19:02:10.039235 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:10.039187 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" podStartSLOduration=2.039172768 podStartE2EDuration="2.039172768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:02:10.036638774 +0000 UTC m=+1665.742527006" watchObservedRunningTime="2026-09-08 19:02:10.039172768 +0000 UTC m=+1665.745061000" Sep 08 19:02:21.027351 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:21.027318 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" Sep 08 19:02:25.036452 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:25.036418 2553 scope.go:117] "RemoveContainer" containerID="149b6ef2e3717abf5e1ddbef93d9daa8d82171e39b1db100d2cba377a403107a" Sep 08 19:02:37.747690 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:37.747654 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm"] Sep 08 19:02:37.751281 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:37.751251 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" Sep 08 19:02:37.753709 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:37.753690 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ee8372\"" Sep 08 19:02:37.760450 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:37.760427 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm"] Sep 08 19:02:37.769578 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:37.769547 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz"] Sep 08 19:02:37.773150 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:37.773131 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" Sep 08 19:02:37.775575 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:37.775554 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ee8372-dockercfg-9hkvh\"" Sep 08 19:02:37.784291 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:37.784270 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz"] Sep 08 19:02:37.900880 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:37.900843 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gq5d\" (UniqueName: \"kubernetes.io/projected/db5d057f-4024-4732-bb1e-e4b0ce6a7db0-kube-api-access-5gq5d\") pod \"payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz\" (UID: \"db5d057f-4024-4732-bb1e-e4b0ce6a7db0\") " pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" Sep 08 19:02:37.901063 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:37.900997 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db5d057f-4024-4732-bb1e-e4b0ce6a7db0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz\" (UID: \"db5d057f-4024-4732-bb1e-e4b0ce6a7db0\") " pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" Sep 08 19:02:37.901063 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:37.901049 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2396fb65-3ade-4653-9873-23c83e921050-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm\" (UID: \"2396fb65-3ade-4653-9873-23c83e921050\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" Sep 08 19:02:37.901163 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:37.901150 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcrsg\" (UniqueName: \"kubernetes.io/projected/2396fb65-3ade-4653-9873-23c83e921050-kube-api-access-hcrsg\") pod \"payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm\" (UID: \"2396fb65-3ade-4653-9873-23c83e921050\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" Sep 08 19:02:38.002209 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:38.002120 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcrsg\" (UniqueName: \"kubernetes.io/projected/2396fb65-3ade-4653-9873-23c83e921050-kube-api-access-hcrsg\") pod \"payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm\" (UID: \"2396fb65-3ade-4653-9873-23c83e921050\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" Sep 08 19:02:38.002209 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:38.002159 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gq5d\" (UniqueName: \"kubernetes.io/projected/db5d057f-4024-4732-bb1e-e4b0ce6a7db0-kube-api-access-5gq5d\") pod \"payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz\" (UID: \"db5d057f-4024-4732-bb1e-e4b0ce6a7db0\") " pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" Sep 08 19:02:38.002471 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:38.002248 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db5d057f-4024-4732-bb1e-e4b0ce6a7db0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz\" (UID: \"db5d057f-4024-4732-bb1e-e4b0ce6a7db0\") " pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" Sep 08 19:02:38.002471 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:38.002284 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2396fb65-3ade-4653-9873-23c83e921050-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm\" (UID: \"2396fb65-3ade-4653-9873-23c83e921050\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" Sep 08 19:02:38.002959 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:38.002942 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2396fb65-3ade-4653-9873-23c83e921050-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm\" (UID: \"2396fb65-3ade-4653-9873-23c83e921050\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" Sep 08 19:02:38.002998 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:38.002954 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db5d057f-4024-4732-bb1e-e4b0ce6a7db0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz\" (UID: \"db5d057f-4024-4732-bb1e-e4b0ce6a7db0\") " pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" Sep 08 19:02:38.010587 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:38.010555 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcrsg\" (UniqueName: \"kubernetes.io/projected/2396fb65-3ade-4653-9873-23c83e921050-kube-api-access-hcrsg\") pod \"payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm\" (UID: \"2396fb65-3ade-4653-9873-23c83e921050\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" Sep 08 19:02:38.010587 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:38.010586 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gq5d\" (UniqueName: \"kubernetes.io/projected/db5d057f-4024-4732-bb1e-e4b0ce6a7db0-kube-api-access-5gq5d\") pod \"payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz\" (UID: \"db5d057f-4024-4732-bb1e-e4b0ce6a7db0\") " pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" Sep 08 19:02:38.063003 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:38.062974 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" Sep 08 19:02:38.084175 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:38.084149 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" Sep 08 19:02:38.342027 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:02:38.341995 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2396fb65_3ade_4653_9873_23c83e921050.slice/crio-72a08cc7e985567fadb30444997e9d6e4219f8d9404da19f807aeb01b8b80759 WatchSource:0}: Error finding container 72a08cc7e985567fadb30444997e9d6e4219f8d9404da19f807aeb01b8b80759: Status 404 returned error can't find the container with id 72a08cc7e985567fadb30444997e9d6e4219f8d9404da19f807aeb01b8b80759 Sep 08 19:02:38.342027 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:38.342016 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm"] Sep 08 19:02:38.372504 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:38.372476 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz"] Sep 08 19:02:38.375546 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:02:38.375523 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5d057f_4024_4732_bb1e_e4b0ce6a7db0.slice/crio-5c2f46dddeb15ce94199a5b7326f831577ed435b095e5ab21d73a0c53abbc900 WatchSource:0}: Error finding container 5c2f46dddeb15ce94199a5b7326f831577ed435b095e5ab21d73a0c53abbc900: Status 404 returned error can't find the container with id 5c2f46dddeb15ce94199a5b7326f831577ed435b095e5ab21d73a0c53abbc900 Sep 08 19:02:39.192948 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:39.192907 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" event={"ID":"2396fb65-3ade-4653-9873-23c83e921050","Type":"ContainerStarted","Data":"3bb2479f407e2a81ea40e40227bff166912b559ec5d51e5ea975f3dbe18cab2a"} Sep 08 19:02:39.193488 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:39.192981 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" event={"ID":"2396fb65-3ade-4653-9873-23c83e921050","Type":"ContainerStarted","Data":"72a08cc7e985567fadb30444997e9d6e4219f8d9404da19f807aeb01b8b80759"} Sep 08 19:02:39.193488 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:39.193015 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" Sep 08 19:02:39.194693 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:39.194666 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" event={"ID":"db5d057f-4024-4732-bb1e-e4b0ce6a7db0","Type":"ContainerStarted","Data":"f125eb9fda7453c9468e1541cf2a864019ae3389bf4d12f418ddf6a30127349e"} Sep 08 19:02:39.194834 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:39.194712 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" Sep 08 19:02:39.194834 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:39.194731 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" event={"ID":"db5d057f-4024-4732-bb1e-e4b0ce6a7db0","Type":"ContainerStarted","Data":"5c2f46dddeb15ce94199a5b7326f831577ed435b095e5ab21d73a0c53abbc900"} Sep 08 19:02:39.221743 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:39.221686 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" podStartSLOduration=2.221672374 podStartE2EDuration="2.221672374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:02:39.218692234 +0000 UTC m=+1694.924580468" watchObservedRunningTime="2026-09-08 19:02:39.221672374 +0000 UTC m=+1694.927560951" Sep 08 19:02:39.239294 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:39.239245 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" podStartSLOduration=2.239226577 podStartE2EDuration="2.239226577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:02:39.236839999 +0000 UTC m=+1694.942728231" watchObservedRunningTime="2026-09-08 19:02:39.239226577 +0000 UTC m=+1694.945114810" Sep 08 19:02:50.202436 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:50.202343 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" Sep 08 19:02:50.202976 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:02:50.202639 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" Sep 08 19:03:03.829670 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:03.829629 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-965746-fkltg"] Sep 08 19:03:03.833224 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:03.833207 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-965746-fkltg" Sep 08 19:03:03.835838 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:03.835822 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 19:03:03.835925 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:03.835836 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 19:03:03.840886 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:03.840856 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-965746-fkltg"] Sep 08 19:03:03.946461 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:03.946424 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6c2450-04ce-4457-b3b2-5468a766705e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-965746-fkltg\" (UID: \"ed6c2450-04ce-4457-b3b2-5468a766705e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-965746-fkltg" Sep 08 19:03:04.047870 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:04.047825 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6c2450-04ce-4457-b3b2-5468a766705e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-965746-fkltg\" (UID: \"ed6c2450-04ce-4457-b3b2-5468a766705e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-965746-fkltg" Sep 08 19:03:04.048511 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:04.048489 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6c2450-04ce-4457-b3b2-5468a766705e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-965746-fkltg\" (UID: \"ed6c2450-04ce-4457-b3b2-5468a766705e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-965746-fkltg" Sep 08 19:03:04.146371 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:04.146339 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-965746-fkltg" Sep 08 19:03:04.423993 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:04.423954 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-965746-fkltg"] Sep 08 19:03:04.425085 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:03:04.425062 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6c2450_04ce_4457_b3b2_5468a766705e.slice/crio-1b4fcdc6d74a24419cc48889fb3ce0076d42fb62a6fd1813ac081aa5219bae71 WatchSource:0}: Error finding container 1b4fcdc6d74a24419cc48889fb3ce0076d42fb62a6fd1813ac081aa5219bae71: Status 404 returned error can't find the container with id 1b4fcdc6d74a24419cc48889fb3ce0076d42fb62a6fd1813ac081aa5219bae71 Sep 08 19:03:05.352486 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:05.352448 2553 generic.go:356] "Generic (PLEG): container finished" podID="ed6c2450-04ce-4457-b3b2-5468a766705e" containerID="e9a4446c56d06362214ae726cbd33e1b874bef1f029bec9bc825fcd2a1e30e48" exitCode=0 Sep 08 19:03:05.352832 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:05.352534 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-965746-fkltg" event={"ID":"ed6c2450-04ce-4457-b3b2-5468a766705e","Type":"ContainerDied","Data":"e9a4446c56d06362214ae726cbd33e1b874bef1f029bec9bc825fcd2a1e30e48"} Sep 08 19:03:05.352832 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:05.352587 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-965746-fkltg" event={"ID":"ed6c2450-04ce-4457-b3b2-5468a766705e","Type":"ContainerStarted","Data":"1b4fcdc6d74a24419cc48889fb3ce0076d42fb62a6fd1813ac081aa5219bae71"} Sep 08 19:03:06.497304 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:06.497283 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-965746-fkltg" Sep 08 19:03:06.571251 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:06.571219 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6c2450-04ce-4457-b3b2-5468a766705e-maas-api-service-ca\") pod \"ed6c2450-04ce-4457-b3b2-5468a766705e\" (UID: \"ed6c2450-04ce-4457-b3b2-5468a766705e\") " Sep 08 19:03:06.571551 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:06.571529 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6c2450-04ce-4457-b3b2-5468a766705e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed6c2450-04ce-4457-b3b2-5468a766705e" (UID: "ed6c2450-04ce-4457-b3b2-5468a766705e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:03:06.672091 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:06.672067 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6c2450-04ce-4457-b3b2-5468a766705e-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:03:07.364901 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:07.364868 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-965746-fkltg" event={"ID":"ed6c2450-04ce-4457-b3b2-5468a766705e","Type":"ContainerDied","Data":"1b4fcdc6d74a24419cc48889fb3ce0076d42fb62a6fd1813ac081aa5219bae71"} Sep 08 19:03:07.365125 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:07.364915 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b4fcdc6d74a24419cc48889fb3ce0076d42fb62a6fd1813ac081aa5219bae71" Sep 08 19:03:07.365125 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:07.364883 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-965746-fkltg" Sep 08 19:03:19.532287 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:19.532183 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw"] Sep 08 19:03:19.532811 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:19.532591 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" podUID="5bfa122d-8806-486c-b2ed-a682095514c1" containerName="payload-processing" containerID="cri-o://be61b3dc184002922aedf42551087717b2746b46827bdbdc5b2b5f9b8bfaaf09" gracePeriod=30 Sep 08 19:03:19.686949 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:19.686912 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-965746-fkltg"] Sep 08 19:03:19.689467 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:19.689439 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-965746-fkltg"] Sep 08 19:03:19.806198 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:19.806171 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" Sep 08 19:03:19.998274 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:19.998242 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrc7k\" (UniqueName: \"kubernetes.io/projected/5bfa122d-8806-486c-b2ed-a682095514c1-kube-api-access-xrc7k\") pod \"5bfa122d-8806-486c-b2ed-a682095514c1\" (UID: \"5bfa122d-8806-486c-b2ed-a682095514c1\") " Sep 08 19:03:19.998491 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:19.998296 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bfa122d-8806-486c-b2ed-a682095514c1-plugins-config-volume\") pod \"5bfa122d-8806-486c-b2ed-a682095514c1\" (UID: \"5bfa122d-8806-486c-b2ed-a682095514c1\") " Sep 08 19:03:19.998693 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:19.998670 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfa122d-8806-486c-b2ed-a682095514c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5bfa122d-8806-486c-b2ed-a682095514c1" (UID: "5bfa122d-8806-486c-b2ed-a682095514c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:03:20.000340 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:20.000320 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfa122d-8806-486c-b2ed-a682095514c1-kube-api-access-xrc7k" (OuterVolumeSpecName: "kube-api-access-xrc7k") pod "5bfa122d-8806-486c-b2ed-a682095514c1" (UID: "5bfa122d-8806-486c-b2ed-a682095514c1"). InnerVolumeSpecName "kube-api-access-xrc7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:03:20.099652 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:20.099582 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrc7k\" (UniqueName: \"kubernetes.io/projected/5bfa122d-8806-486c-b2ed-a682095514c1-kube-api-access-xrc7k\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:03:20.099652 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:20.099611 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bfa122d-8806-486c-b2ed-a682095514c1-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:03:20.446111 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:20.446073 2553 generic.go:356] "Generic (PLEG): container finished" podID="5bfa122d-8806-486c-b2ed-a682095514c1" containerID="be61b3dc184002922aedf42551087717b2746b46827bdbdc5b2b5f9b8bfaaf09" exitCode=0 Sep 08 19:03:20.446111 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:20.446109 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" Sep 08 19:03:20.446326 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:20.446159 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" event={"ID":"5bfa122d-8806-486c-b2ed-a682095514c1","Type":"ContainerDied","Data":"be61b3dc184002922aedf42551087717b2746b46827bdbdc5b2b5f9b8bfaaf09"} Sep 08 19:03:20.446326 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:20.446209 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw" event={"ID":"5bfa122d-8806-486c-b2ed-a682095514c1","Type":"ContainerDied","Data":"c88cb2d9a00e9b66d369d0e52cbce8e48f93b3d33d42a171368cbdfd459c4b93"} Sep 08 19:03:20.446326 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:20.446236 2553 scope.go:117] "RemoveContainer" containerID="be61b3dc184002922aedf42551087717b2746b46827bdbdc5b2b5f9b8bfaaf09" Sep 08 19:03:20.455711 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:20.455694 2553 scope.go:117] "RemoveContainer" containerID="be61b3dc184002922aedf42551087717b2746b46827bdbdc5b2b5f9b8bfaaf09" Sep 08 19:03:20.455975 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:03:20.455958 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be61b3dc184002922aedf42551087717b2746b46827bdbdc5b2b5f9b8bfaaf09\": container with ID starting with be61b3dc184002922aedf42551087717b2746b46827bdbdc5b2b5f9b8bfaaf09 not found: ID does not exist" containerID="be61b3dc184002922aedf42551087717b2746b46827bdbdc5b2b5f9b8bfaaf09" Sep 08 19:03:20.456024 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:20.455984 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be61b3dc184002922aedf42551087717b2746b46827bdbdc5b2b5f9b8bfaaf09"} err="failed to get container status \"be61b3dc184002922aedf42551087717b2746b46827bdbdc5b2b5f9b8bfaaf09\": rpc error: code = NotFound desc = could not find container \"be61b3dc184002922aedf42551087717b2746b46827bdbdc5b2b5f9b8bfaaf09\": container with ID starting with be61b3dc184002922aedf42551087717b2746b46827bdbdc5b2b5f9b8bfaaf09 not found: ID does not exist" Sep 08 19:03:20.468250 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:20.468228 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw"] Sep 08 19:03:20.473224 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:20.473198 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-965746-dc45866d5-j5wlw"] Sep 08 19:03:20.745878 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:20.745793 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bfa122d-8806-486c-b2ed-a682095514c1" path="/var/lib/kubelet/pods/5bfa122d-8806-486c-b2ed-a682095514c1/volumes" Sep 08 19:03:20.746221 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:20.746126 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed6c2450-04ce-4457-b3b2-5468a766705e" path="/var/lib/kubelet/pods/ed6c2450-04ce-4457-b3b2-5468a766705e/volumes" Sep 08 19:03:25.058103 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:25.058065 2553 scope.go:117] "RemoveContainer" containerID="9ce5d3e2b43ed5b27764547c1b9163f8a21f03bb1872e80f440f0b7d422f4800" Sep 08 19:03:25.066686 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:25.066665 2553 scope.go:117] "RemoveContainer" containerID="6cf31c058700fddd8b5788059cee06d8b28cdf79681766ad0c6ae1a67eb61d3a" Sep 08 19:03:26.527582 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:26.527551 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb"] Sep 08 19:03:26.527945 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:26.527938 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6c2450-04ce-4457-b3b2-5468a766705e" containerName="revoke-keys" Sep 08 19:03:26.527985 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:26.527949 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6c2450-04ce-4457-b3b2-5468a766705e" containerName="revoke-keys" Sep 08 19:03:26.527985 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:26.527967 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bfa122d-8806-486c-b2ed-a682095514c1" containerName="payload-processing" Sep 08 19:03:26.527985 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:26.527973 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bfa122d-8806-486c-b2ed-a682095514c1" containerName="payload-processing" Sep 08 19:03:26.528084 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:26.528038 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bfa122d-8806-486c-b2ed-a682095514c1" containerName="payload-processing" Sep 08 19:03:26.528084 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:26.528051 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed6c2450-04ce-4457-b3b2-5468a766705e" containerName="revoke-keys" Sep 08 19:03:26.532908 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:26.532891 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb" Sep 08 19:03:26.536122 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:26.536094 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 19:03:26.536263 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:26.536163 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 19:03:26.539072 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:26.539049 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb"] Sep 08 19:03:26.555668 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:26.555642 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb\" (UID: \"eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb" Sep 08 19:03:26.656318 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:26.656287 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb\" (UID: \"eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb" Sep 08 19:03:26.656941 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:26.656920 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb\" (UID: \"eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb" Sep 08 19:03:26.844541 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:26.844462 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb" Sep 08 19:03:27.117634 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:27.117604 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb"] Sep 08 19:03:27.118999 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:03:27.118956 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1faddf_2892_4ea2_bd0f_91dbbdeb54b2.slice/crio-53aec03879457dadd4fbfe3bc6352e1ac106a2e842ef09a82ee1e17a95194107 WatchSource:0}: Error finding container 53aec03879457dadd4fbfe3bc6352e1ac106a2e842ef09a82ee1e17a95194107: Status 404 returned error can't find the container with id 53aec03879457dadd4fbfe3bc6352e1ac106a2e842ef09a82ee1e17a95194107 Sep 08 19:03:27.491071 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:27.491043 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb" event={"ID":"eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2","Type":"ContainerStarted","Data":"28a1d4bdfdd311f6638fd7331566453d0cf3d1676faeeb439b827be446129470"} Sep 08 19:03:27.491198 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:27.491092 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb" event={"ID":"eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2","Type":"ContainerStarted","Data":"53aec03879457dadd4fbfe3bc6352e1ac106a2e842ef09a82ee1e17a95194107"} Sep 08 19:03:27.507024 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:27.506982 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb" podStartSLOduration=1.5069669989999999 podStartE2EDuration="1.506966999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:03:27.505648142 +0000 UTC m=+1743.211536376" watchObservedRunningTime="2026-09-08 19:03:27.506966999 +0000 UTC m=+1743.212855230" Sep 08 19:03:28.497693 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:28.497661 2553 generic.go:356] "Generic (PLEG): container finished" podID="eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2" containerID="28a1d4bdfdd311f6638fd7331566453d0cf3d1676faeeb439b827be446129470" exitCode=0 Sep 08 19:03:28.498034 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:28.497750 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb" event={"ID":"eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2","Type":"ContainerDied","Data":"28a1d4bdfdd311f6638fd7331566453d0cf3d1676faeeb439b827be446129470"} Sep 08 19:03:29.642766 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:29.642738 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb" Sep 08 19:03:29.685355 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:29.685328 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2-maas-api-service-ca\") pod \"eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2\" (UID: \"eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2\") " Sep 08 19:03:29.685662 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:29.685640 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2" (UID: "eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:03:29.786844 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:29.786780 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:03:30.511150 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:30.511112 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb" event={"ID":"eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2","Type":"ContainerDied","Data":"53aec03879457dadd4fbfe3bc6352e1ac106a2e842ef09a82ee1e17a95194107"} Sep 08 19:03:30.511150 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:30.511142 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb" Sep 08 19:03:30.511445 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:30.511186 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53aec03879457dadd4fbfe3bc6352e1ac106a2e842ef09a82ee1e17a95194107" Sep 08 19:03:42.031973 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.031941 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz"] Sep 08 19:03:42.033180 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.033142 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" podUID="db5d057f-4024-4732-bb1e-e4b0ce6a7db0" containerName="payload-processing" containerID="cri-o://f125eb9fda7453c9468e1541cf2a864019ae3389bf4d12f418ddf6a30127349e" gracePeriod=30 Sep 08 19:03:42.038523 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.038497 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm"] Sep 08 19:03:42.038735 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.038706 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" podUID="2396fb65-3ade-4653-9873-23c83e921050" containerName="payload-pre-processing" containerID="cri-o://3bb2479f407e2a81ea40e40227bff166912b559ec5d51e5ea975f3dbe18cab2a" gracePeriod=30 Sep 08 19:03:42.181306 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.181253 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb"] Sep 08 19:03:42.183790 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.183761 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ee8372-gw7hb"] Sep 08 19:03:42.316079 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.316056 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" Sep 08 19:03:42.318779 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.318760 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" Sep 08 19:03:42.397944 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.397915 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2396fb65-3ade-4653-9873-23c83e921050-plugins-config-volume\") pod \"2396fb65-3ade-4653-9873-23c83e921050\" (UID: \"2396fb65-3ade-4653-9873-23c83e921050\") " Sep 08 19:03:42.398138 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.397969 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcrsg\" (UniqueName: \"kubernetes.io/projected/2396fb65-3ade-4653-9873-23c83e921050-kube-api-access-hcrsg\") pod \"2396fb65-3ade-4653-9873-23c83e921050\" (UID: \"2396fb65-3ade-4653-9873-23c83e921050\") " Sep 08 19:03:42.398138 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.398000 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db5d057f-4024-4732-bb1e-e4b0ce6a7db0-plugins-config-volume\") pod \"db5d057f-4024-4732-bb1e-e4b0ce6a7db0\" (UID: \"db5d057f-4024-4732-bb1e-e4b0ce6a7db0\") " Sep 08 19:03:42.398138 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.398048 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gq5d\" (UniqueName: \"kubernetes.io/projected/db5d057f-4024-4732-bb1e-e4b0ce6a7db0-kube-api-access-5gq5d\") pod \"db5d057f-4024-4732-bb1e-e4b0ce6a7db0\" (UID: \"db5d057f-4024-4732-bb1e-e4b0ce6a7db0\") " Sep 08 19:03:42.398415 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.398391 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2396fb65-3ade-4653-9873-23c83e921050-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2396fb65-3ade-4653-9873-23c83e921050" (UID: "2396fb65-3ade-4653-9873-23c83e921050"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:03:42.398472 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.398407 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5d057f-4024-4732-bb1e-e4b0ce6a7db0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "db5d057f-4024-4732-bb1e-e4b0ce6a7db0" (UID: "db5d057f-4024-4732-bb1e-e4b0ce6a7db0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:03:42.400177 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.400144 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5d057f-4024-4732-bb1e-e4b0ce6a7db0-kube-api-access-5gq5d" (OuterVolumeSpecName: "kube-api-access-5gq5d") pod "db5d057f-4024-4732-bb1e-e4b0ce6a7db0" (UID: "db5d057f-4024-4732-bb1e-e4b0ce6a7db0"). InnerVolumeSpecName "kube-api-access-5gq5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:03:42.400177 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.400164 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2396fb65-3ade-4653-9873-23c83e921050-kube-api-access-hcrsg" (OuterVolumeSpecName: "kube-api-access-hcrsg") pod "2396fb65-3ade-4653-9873-23c83e921050" (UID: "2396fb65-3ade-4653-9873-23c83e921050"). InnerVolumeSpecName "kube-api-access-hcrsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:03:42.499355 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.499328 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2396fb65-3ade-4653-9873-23c83e921050-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:03:42.499355 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.499352 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcrsg\" (UniqueName: \"kubernetes.io/projected/2396fb65-3ade-4653-9873-23c83e921050-kube-api-access-hcrsg\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:03:42.499549 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.499386 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db5d057f-4024-4732-bb1e-e4b0ce6a7db0-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:03:42.499549 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.499396 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gq5d\" (UniqueName: \"kubernetes.io/projected/db5d057f-4024-4732-bb1e-e4b0ce6a7db0-kube-api-access-5gq5d\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:03:42.579453 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.579373 2553 generic.go:356] "Generic (PLEG): container finished" podID="2396fb65-3ade-4653-9873-23c83e921050" containerID="3bb2479f407e2a81ea40e40227bff166912b559ec5d51e5ea975f3dbe18cab2a" exitCode=0 Sep 08 19:03:42.579453 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.579425 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" Sep 08 19:03:42.579633 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.579455 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" event={"ID":"2396fb65-3ade-4653-9873-23c83e921050","Type":"ContainerDied","Data":"3bb2479f407e2a81ea40e40227bff166912b559ec5d51e5ea975f3dbe18cab2a"} Sep 08 19:03:42.579633 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.579509 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm" event={"ID":"2396fb65-3ade-4653-9873-23c83e921050","Type":"ContainerDied","Data":"72a08cc7e985567fadb30444997e9d6e4219f8d9404da19f807aeb01b8b80759"} Sep 08 19:03:42.579633 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.579527 2553 scope.go:117] "RemoveContainer" containerID="3bb2479f407e2a81ea40e40227bff166912b559ec5d51e5ea975f3dbe18cab2a" Sep 08 19:03:42.580871 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.580852 2553 generic.go:356] "Generic (PLEG): container finished" podID="db5d057f-4024-4732-bb1e-e4b0ce6a7db0" containerID="f125eb9fda7453c9468e1541cf2a864019ae3389bf4d12f418ddf6a30127349e" exitCode=0 Sep 08 19:03:42.580958 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.580886 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" event={"ID":"db5d057f-4024-4732-bb1e-e4b0ce6a7db0","Type":"ContainerDied","Data":"f125eb9fda7453c9468e1541cf2a864019ae3389bf4d12f418ddf6a30127349e"} Sep 08 19:03:42.580958 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.580906 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" Sep 08 19:03:42.580958 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.580940 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz" event={"ID":"db5d057f-4024-4732-bb1e-e4b0ce6a7db0","Type":"ContainerDied","Data":"5c2f46dddeb15ce94199a5b7326f831577ed435b095e5ab21d73a0c53abbc900"} Sep 08 19:03:42.588914 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.588903 2553 scope.go:117] "RemoveContainer" containerID="3bb2479f407e2a81ea40e40227bff166912b559ec5d51e5ea975f3dbe18cab2a" Sep 08 19:03:42.589209 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:03:42.589189 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb2479f407e2a81ea40e40227bff166912b559ec5d51e5ea975f3dbe18cab2a\": container with ID starting with 3bb2479f407e2a81ea40e40227bff166912b559ec5d51e5ea975f3dbe18cab2a not found: ID does not exist" containerID="3bb2479f407e2a81ea40e40227bff166912b559ec5d51e5ea975f3dbe18cab2a" Sep 08 19:03:42.589286 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.589214 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb2479f407e2a81ea40e40227bff166912b559ec5d51e5ea975f3dbe18cab2a"} err="failed to get container status \"3bb2479f407e2a81ea40e40227bff166912b559ec5d51e5ea975f3dbe18cab2a\": rpc error: code = NotFound desc = could not find container \"3bb2479f407e2a81ea40e40227bff166912b559ec5d51e5ea975f3dbe18cab2a\": container with ID starting with 3bb2479f407e2a81ea40e40227bff166912b559ec5d51e5ea975f3dbe18cab2a not found: ID does not exist" Sep 08 19:03:42.589286 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.589230 2553 scope.go:117] "RemoveContainer" containerID="f125eb9fda7453c9468e1541cf2a864019ae3389bf4d12f418ddf6a30127349e" Sep 08 19:03:42.597544 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.597529 2553 scope.go:117] "RemoveContainer" containerID="f125eb9fda7453c9468e1541cf2a864019ae3389bf4d12f418ddf6a30127349e" Sep 08 19:03:42.597781 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:03:42.597766 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f125eb9fda7453c9468e1541cf2a864019ae3389bf4d12f418ddf6a30127349e\": container with ID starting with f125eb9fda7453c9468e1541cf2a864019ae3389bf4d12f418ddf6a30127349e not found: ID does not exist" containerID="f125eb9fda7453c9468e1541cf2a864019ae3389bf4d12f418ddf6a30127349e" Sep 08 19:03:42.597852 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.597786 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f125eb9fda7453c9468e1541cf2a864019ae3389bf4d12f418ddf6a30127349e"} err="failed to get container status \"f125eb9fda7453c9468e1541cf2a864019ae3389bf4d12f418ddf6a30127349e\": rpc error: code = NotFound desc = could not find container \"f125eb9fda7453c9468e1541cf2a864019ae3389bf4d12f418ddf6a30127349e\": container with ID starting with f125eb9fda7453c9468e1541cf2a864019ae3389bf4d12f418ddf6a30127349e not found: ID does not exist" Sep 08 19:03:42.606423 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.606402 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm"] Sep 08 19:03:42.609113 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.609093 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ee8372-79997559bb-lhksm"] Sep 08 19:03:42.617983 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.617958 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz"] Sep 08 19:03:42.623660 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.623640 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ee8372-7b47d8f7cc-7jplz"] Sep 08 19:03:42.745561 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.745537 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2396fb65-3ade-4653-9873-23c83e921050" path="/var/lib/kubelet/pods/2396fb65-3ade-4653-9873-23c83e921050/volumes" Sep 08 19:03:42.745918 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.745905 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db5d057f-4024-4732-bb1e-e4b0ce6a7db0" path="/var/lib/kubelet/pods/db5d057f-4024-4732-bb1e-e4b0ce6a7db0/volumes" Sep 08 19:03:42.746189 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:42.746178 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2" path="/var/lib/kubelet/pods/eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2/volumes" Sep 08 19:03:49.254783 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.254744 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455"] Sep 08 19:03:49.255172 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.255158 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db5d057f-4024-4732-bb1e-e4b0ce6a7db0" containerName="payload-processing" Sep 08 19:03:49.255233 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.255173 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="db5d057f-4024-4732-bb1e-e4b0ce6a7db0" containerName="payload-processing" Sep 08 19:03:49.255233 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.255184 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2396fb65-3ade-4653-9873-23c83e921050" containerName="payload-pre-processing" Sep 08 19:03:49.255233 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.255190 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2396fb65-3ade-4653-9873-23c83e921050" containerName="payload-pre-processing" Sep 08 19:03:49.255233 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.255205 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2" containerName="revoke-keys" Sep 08 19:03:49.255233 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.255211 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2" containerName="revoke-keys" Sep 08 19:03:49.255408 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.255278 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb1faddf-2892-4ea2-bd0f-91dbbdeb54b2" containerName="revoke-keys" Sep 08 19:03:49.255408 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.255286 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2396fb65-3ade-4653-9873-23c83e921050" containerName="payload-pre-processing" Sep 08 19:03:49.255408 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.255294 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="db5d057f-4024-4732-bb1e-e4b0ce6a7db0" containerName="payload-processing" Sep 08 19:03:49.260033 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.260015 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455" Sep 08 19:03:49.263600 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.263442 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 19:03:49.263600 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.263442 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 19:03:49.269010 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.268979 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455"] Sep 08 19:03:49.371077 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.371040 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38aedbe-aeec-4e3b-a369-a141c4c0fb4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455\" (UID: \"a38aedbe-aeec-4e3b-a369-a141c4c0fb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455" Sep 08 19:03:49.473188 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.472518 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38aedbe-aeec-4e3b-a369-a141c4c0fb4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455\" (UID: \"a38aedbe-aeec-4e3b-a369-a141c4c0fb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455" Sep 08 19:03:49.473507 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.473481 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38aedbe-aeec-4e3b-a369-a141c4c0fb4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455\" (UID: \"a38aedbe-aeec-4e3b-a369-a141c4c0fb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455" Sep 08 19:03:49.585637 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.585548 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455" Sep 08 19:03:49.856961 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:49.856936 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455"] Sep 08 19:03:49.858333 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:03:49.858308 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38aedbe_aeec_4e3b_a369_a141c4c0fb4c.slice/crio-07bb75e96df47429810a6fd89945f625d9d0ad503ac0c9c060d1d9bb50859886 WatchSource:0}: Error finding container 07bb75e96df47429810a6fd89945f625d9d0ad503ac0c9c060d1d9bb50859886: Status 404 returned error can't find the container with id 07bb75e96df47429810a6fd89945f625d9d0ad503ac0c9c060d1d9bb50859886 Sep 08 19:03:50.631170 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:50.631141 2553 generic.go:356] "Generic (PLEG): container finished" podID="a38aedbe-aeec-4e3b-a369-a141c4c0fb4c" containerID="5575cfdd6c4d3d910cf8ff6952e28fc79acde7c12013f9e054f6e88fe4da02c2" exitCode=0 Sep 08 19:03:50.631547 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:50.631207 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455" event={"ID":"a38aedbe-aeec-4e3b-a369-a141c4c0fb4c","Type":"ContainerDied","Data":"5575cfdd6c4d3d910cf8ff6952e28fc79acde7c12013f9e054f6e88fe4da02c2"} Sep 08 19:03:50.631547 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:50.631244 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455" event={"ID":"a38aedbe-aeec-4e3b-a369-a141c4c0fb4c","Type":"ContainerStarted","Data":"07bb75e96df47429810a6fd89945f625d9d0ad503ac0c9c060d1d9bb50859886"} Sep 08 19:03:51.765608 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:51.765582 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455" Sep 08 19:03:51.897358 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:51.897276 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38aedbe-aeec-4e3b-a369-a141c4c0fb4c-maas-api-service-ca\") pod \"a38aedbe-aeec-4e3b-a369-a141c4c0fb4c\" (UID: \"a38aedbe-aeec-4e3b-a369-a141c4c0fb4c\") " Sep 08 19:03:51.897646 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:51.897624 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38aedbe-aeec-4e3b-a369-a141c4c0fb4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a38aedbe-aeec-4e3b-a369-a141c4c0fb4c" (UID: "a38aedbe-aeec-4e3b-a369-a141c4c0fb4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:03:51.998123 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:51.998089 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a38aedbe-aeec-4e3b-a369-a141c4c0fb4c-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:03:52.644468 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:52.644430 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455" event={"ID":"a38aedbe-aeec-4e3b-a369-a141c4c0fb4c","Type":"ContainerDied","Data":"07bb75e96df47429810a6fd89945f625d9d0ad503ac0c9c060d1d9bb50859886"} Sep 08 19:03:52.644468 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:52.644466 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455" Sep 08 19:03:52.644721 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:03:52.644489 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07bb75e96df47429810a6fd89945f625d9d0ad503ac0c9c060d1d9bb50859886" Sep 08 19:04:00.106598 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.106565 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5"] Sep 08 19:04:00.107195 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.106805 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" podUID="a9588f03-8418-4ff4-8001-743797e1cd5a" containerName="payload-processing" containerID="cri-o://759338af8d0e7a9c38be6d7bf8ee35243d2508920104117f0f9fbc5fb042e556" gracePeriod=30 Sep 08 19:04:00.277062 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.277032 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455"] Sep 08 19:04:00.281658 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.281630 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-093570-g7455"] Sep 08 19:04:00.365591 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.365530 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" Sep 08 19:04:00.379170 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.379149 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9588f03-8418-4ff4-8001-743797e1cd5a-plugins-config-volume\") pod \"a9588f03-8418-4ff4-8001-743797e1cd5a\" (UID: \"a9588f03-8418-4ff4-8001-743797e1cd5a\") " Sep 08 19:04:00.379263 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.379189 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhzmc\" (UniqueName: \"kubernetes.io/projected/a9588f03-8418-4ff4-8001-743797e1cd5a-kube-api-access-hhzmc\") pod \"a9588f03-8418-4ff4-8001-743797e1cd5a\" (UID: \"a9588f03-8418-4ff4-8001-743797e1cd5a\") " Sep 08 19:04:00.379513 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.379488 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9588f03-8418-4ff4-8001-743797e1cd5a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9588f03-8418-4ff4-8001-743797e1cd5a" (UID: "a9588f03-8418-4ff4-8001-743797e1cd5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:04:00.381127 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.381106 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9588f03-8418-4ff4-8001-743797e1cd5a-kube-api-access-hhzmc" (OuterVolumeSpecName: "kube-api-access-hhzmc") pod "a9588f03-8418-4ff4-8001-743797e1cd5a" (UID: "a9588f03-8418-4ff4-8001-743797e1cd5a"). InnerVolumeSpecName "kube-api-access-hhzmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:04:00.486605 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.480189 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9588f03-8418-4ff4-8001-743797e1cd5a-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:04:00.486605 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.486298 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhzmc\" (UniqueName: \"kubernetes.io/projected/a9588f03-8418-4ff4-8001-743797e1cd5a-kube-api-access-hhzmc\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:04:00.696104 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.696062 2553 generic.go:356] "Generic (PLEG): container finished" podID="a9588f03-8418-4ff4-8001-743797e1cd5a" containerID="759338af8d0e7a9c38be6d7bf8ee35243d2508920104117f0f9fbc5fb042e556" exitCode=0 Sep 08 19:04:00.696279 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.696120 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" Sep 08 19:04:00.696279 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.696147 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" event={"ID":"a9588f03-8418-4ff4-8001-743797e1cd5a","Type":"ContainerDied","Data":"759338af8d0e7a9c38be6d7bf8ee35243d2508920104117f0f9fbc5fb042e556"} Sep 08 19:04:00.696279 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.696242 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5" event={"ID":"a9588f03-8418-4ff4-8001-743797e1cd5a","Type":"ContainerDied","Data":"8abb6f570d4942e1ae71d4b0908687c865ce60fc74134e500b9fdc90e7144f09"} Sep 08 19:04:00.696279 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.696267 2553 scope.go:117] "RemoveContainer" containerID="759338af8d0e7a9c38be6d7bf8ee35243d2508920104117f0f9fbc5fb042e556" Sep 08 19:04:00.705935 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.705917 2553 scope.go:117] "RemoveContainer" containerID="759338af8d0e7a9c38be6d7bf8ee35243d2508920104117f0f9fbc5fb042e556" Sep 08 19:04:00.706193 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:04:00.706173 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"759338af8d0e7a9c38be6d7bf8ee35243d2508920104117f0f9fbc5fb042e556\": container with ID starting with 759338af8d0e7a9c38be6d7bf8ee35243d2508920104117f0f9fbc5fb042e556 not found: ID does not exist" containerID="759338af8d0e7a9c38be6d7bf8ee35243d2508920104117f0f9fbc5fb042e556" Sep 08 19:04:00.706259 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.706203 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759338af8d0e7a9c38be6d7bf8ee35243d2508920104117f0f9fbc5fb042e556"} err="failed to get container status \"759338af8d0e7a9c38be6d7bf8ee35243d2508920104117f0f9fbc5fb042e556\": rpc error: code = NotFound desc = could not find container \"759338af8d0e7a9c38be6d7bf8ee35243d2508920104117f0f9fbc5fb042e556\": container with ID starting with 759338af8d0e7a9c38be6d7bf8ee35243d2508920104117f0f9fbc5fb042e556 not found: ID does not exist" Sep 08 19:04:00.718579 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.718554 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5"] Sep 08 19:04:00.721091 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.721072 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-093570-5df66bf465-m59d5"] Sep 08 19:04:00.749175 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.749137 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a38aedbe-aeec-4e3b-a369-a141c4c0fb4c" path="/var/lib/kubelet/pods/a38aedbe-aeec-4e3b-a369-a141c4c0fb4c/volumes" Sep 08 19:04:00.749745 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:00.749718 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9588f03-8418-4ff4-8001-743797e1cd5a" path="/var/lib/kubelet/pods/a9588f03-8418-4ff4-8001-743797e1cd5a/volumes" Sep 08 19:04:04.358758 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.358724 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr"] Sep 08 19:04:04.359291 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.359271 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9588f03-8418-4ff4-8001-743797e1cd5a" containerName="payload-processing" Sep 08 19:04:04.359291 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.359292 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9588f03-8418-4ff4-8001-743797e1cd5a" containerName="payload-processing" Sep 08 19:04:04.359468 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.359314 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a38aedbe-aeec-4e3b-a369-a141c4c0fb4c" containerName="revoke-keys" Sep 08 19:04:04.359468 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.359320 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a38aedbe-aeec-4e3b-a369-a141c4c0fb4c" containerName="revoke-keys" Sep 08 19:04:04.359468 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.359413 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9588f03-8418-4ff4-8001-743797e1cd5a" containerName="payload-processing" Sep 08 19:04:04.359468 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.359427 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a38aedbe-aeec-4e3b-a369-a141c4c0fb4c" containerName="revoke-keys" Sep 08 19:04:04.364240 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.364217 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" Sep 08 19:04:04.368461 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.368441 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e77525-dockercfg-wzvfs\"" Sep 08 19:04:04.368461 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.368454 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e77525\"" Sep 08 19:04:04.377137 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.377112 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr"] Sep 08 19:04:04.407555 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.407522 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6"] Sep 08 19:04:04.411052 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.411036 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" Sep 08 19:04:04.421350 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.421326 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6"] Sep 08 19:04:04.423477 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.423450 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25813327-9a1c-4b26-8417-58aeba664091-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr\" (UID: \"25813327-9a1c-4b26-8417-58aeba664091\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" Sep 08 19:04:04.423596 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.423559 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59h97\" (UniqueName: \"kubernetes.io/projected/25813327-9a1c-4b26-8417-58aeba664091-kube-api-access-59h97\") pod \"payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr\" (UID: \"25813327-9a1c-4b26-8417-58aeba664091\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" Sep 08 19:04:04.525414 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.525357 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5h6l\" (UniqueName: \"kubernetes.io/projected/35bd4061-c9b0-4fd4-a1cf-ec835b614d44-kube-api-access-n5h6l\") pod \"payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6\" (UID: \"35bd4061-c9b0-4fd4-a1cf-ec835b614d44\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" Sep 08 19:04:04.525582 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.525451 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25813327-9a1c-4b26-8417-58aeba664091-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr\" (UID: \"25813327-9a1c-4b26-8417-58aeba664091\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" Sep 08 19:04:04.525582 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.525492 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bd4061-c9b0-4fd4-a1cf-ec835b614d44-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6\" (UID: \"35bd4061-c9b0-4fd4-a1cf-ec835b614d44\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" Sep 08 19:04:04.525582 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.525548 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59h97\" (UniqueName: \"kubernetes.io/projected/25813327-9a1c-4b26-8417-58aeba664091-kube-api-access-59h97\") pod \"payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr\" (UID: \"25813327-9a1c-4b26-8417-58aeba664091\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" Sep 08 19:04:04.526153 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.526132 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25813327-9a1c-4b26-8417-58aeba664091-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr\" (UID: \"25813327-9a1c-4b26-8417-58aeba664091\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" Sep 08 19:04:04.533490 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.533466 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59h97\" (UniqueName: \"kubernetes.io/projected/25813327-9a1c-4b26-8417-58aeba664091-kube-api-access-59h97\") pod \"payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr\" (UID: \"25813327-9a1c-4b26-8417-58aeba664091\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" Sep 08 19:04:04.626964 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.626937 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bd4061-c9b0-4fd4-a1cf-ec835b614d44-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6\" (UID: \"35bd4061-c9b0-4fd4-a1cf-ec835b614d44\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" Sep 08 19:04:04.627121 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.627035 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5h6l\" (UniqueName: \"kubernetes.io/projected/35bd4061-c9b0-4fd4-a1cf-ec835b614d44-kube-api-access-n5h6l\") pod \"payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6\" (UID: \"35bd4061-c9b0-4fd4-a1cf-ec835b614d44\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" Sep 08 19:04:04.627671 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.627645 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bd4061-c9b0-4fd4-a1cf-ec835b614d44-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6\" (UID: \"35bd4061-c9b0-4fd4-a1cf-ec835b614d44\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" Sep 08 19:04:04.635644 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.635614 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5h6l\" (UniqueName: \"kubernetes.io/projected/35bd4061-c9b0-4fd4-a1cf-ec835b614d44-kube-api-access-n5h6l\") pod \"payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6\" (UID: \"35bd4061-c9b0-4fd4-a1cf-ec835b614d44\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" Sep 08 19:04:04.675341 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.675312 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" Sep 08 19:04:04.722179 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.722148 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" Sep 08 19:04:04.950231 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:04.950198 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr"] Sep 08 19:04:04.951547 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:04:04.951519 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25813327_9a1c_4b26_8417_58aeba664091.slice/crio-3d044997f4ad033bd26613fe84ba45d3f397dba64909227ff5ab78ba6db87830 WatchSource:0}: Error finding container 3d044997f4ad033bd26613fe84ba45d3f397dba64909227ff5ab78ba6db87830: Status 404 returned error can't find the container with id 3d044997f4ad033bd26613fe84ba45d3f397dba64909227ff5ab78ba6db87830 Sep 08 19:04:05.020525 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:05.020497 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6"] Sep 08 19:04:05.023452 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:04:05.023426 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bd4061_c9b0_4fd4_a1cf_ec835b614d44.slice/crio-f42580e8d17e2e2ace321a51b259a7d70cffe236e85521be1c41be7851cc3954 WatchSource:0}: Error finding container f42580e8d17e2e2ace321a51b259a7d70cffe236e85521be1c41be7851cc3954: Status 404 returned error can't find the container with id f42580e8d17e2e2ace321a51b259a7d70cffe236e85521be1c41be7851cc3954 Sep 08 19:04:05.728411 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:05.728358 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" event={"ID":"25813327-9a1c-4b26-8417-58aeba664091","Type":"ContainerStarted","Data":"91e008a429b9e89965376f4032927df6ffc5402b26838c4454a2730a7fd6343b"} Sep 08 19:04:05.728911 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:05.728447 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" Sep 08 19:04:05.728911 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:05.728474 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" event={"ID":"25813327-9a1c-4b26-8417-58aeba664091","Type":"ContainerStarted","Data":"3d044997f4ad033bd26613fe84ba45d3f397dba64909227ff5ab78ba6db87830"} Sep 08 19:04:05.730030 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:05.730004 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" event={"ID":"35bd4061-c9b0-4fd4-a1cf-ec835b614d44","Type":"ContainerStarted","Data":"b4c8f83bad49085565f0ffd7f6f1fdbf515316ec7f05a70eb2eb74e531867fa9"} Sep 08 19:04:05.730160 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:05.730054 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" event={"ID":"35bd4061-c9b0-4fd4-a1cf-ec835b614d44","Type":"ContainerStarted","Data":"f42580e8d17e2e2ace321a51b259a7d70cffe236e85521be1c41be7851cc3954"} Sep 08 19:04:05.730160 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:05.730102 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" Sep 08 19:04:05.744392 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:05.744321 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" podStartSLOduration=1.744309217 podStartE2EDuration="1.744309217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:04:05.744045315 +0000 UTC m=+1781.449933557" watchObservedRunningTime="2026-09-08 19:04:05.744309217 +0000 UTC m=+1781.450197449" Sep 08 19:04:05.762792 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:05.762755 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" podStartSLOduration=1.762742647 podStartE2EDuration="1.762742647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:04:05.76084209 +0000 UTC m=+1781.466730322" watchObservedRunningTime="2026-09-08 19:04:05.762742647 +0000 UTC m=+1781.468630935" Sep 08 19:04:07.114904 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:07.114867 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v"] Sep 08 19:04:07.118779 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:07.118755 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v" Sep 08 19:04:07.122657 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:07.122637 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 19:04:07.123249 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:07.123227 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 19:04:07.136485 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:07.136462 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v"] Sep 08 19:04:07.155300 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:07.155110 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca584214-2809-413c-9ccf-e8f35a13fc4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v\" (UID: \"ca584214-2809-413c-9ccf-e8f35a13fc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v" Sep 08 19:04:07.256325 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:07.256289 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca584214-2809-413c-9ccf-e8f35a13fc4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v\" (UID: \"ca584214-2809-413c-9ccf-e8f35a13fc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v" Sep 08 19:04:07.256970 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:07.256949 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca584214-2809-413c-9ccf-e8f35a13fc4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v\" (UID: \"ca584214-2809-413c-9ccf-e8f35a13fc4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v" Sep 08 19:04:07.429721 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:07.429674 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v" Sep 08 19:04:07.703347 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:07.703316 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v"] Sep 08 19:04:07.705111 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:04:07.705085 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca584214_2809_413c_9ccf_e8f35a13fc4a.slice/crio-7f9a7ea11145e436e837d65fb6057151fbefa6507ab28422c3d9c5f2a653cbcb WatchSource:0}: Error finding container 7f9a7ea11145e436e837d65fb6057151fbefa6507ab28422c3d9c5f2a653cbcb: Status 404 returned error can't find the container with id 7f9a7ea11145e436e837d65fb6057151fbefa6507ab28422c3d9c5f2a653cbcb Sep 08 19:04:07.746492 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:07.746464 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v" event={"ID":"ca584214-2809-413c-9ccf-e8f35a13fc4a","Type":"ContainerStarted","Data":"7f9a7ea11145e436e837d65fb6057151fbefa6507ab28422c3d9c5f2a653cbcb"} Sep 08 19:04:08.752830 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:08.752796 2553 generic.go:356] "Generic (PLEG): container finished" podID="ca584214-2809-413c-9ccf-e8f35a13fc4a" containerID="c1cab08259c29b645d6f0475e6c93780fc4bbede6c2063dabc88c46ed1300515" exitCode=0 Sep 08 19:04:08.753182 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:08.752880 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v" event={"ID":"ca584214-2809-413c-9ccf-e8f35a13fc4a","Type":"ContainerDied","Data":"c1cab08259c29b645d6f0475e6c93780fc4bbede6c2063dabc88c46ed1300515"} Sep 08 19:04:09.896703 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:09.896678 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v" Sep 08 19:04:09.982755 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:09.982711 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca584214-2809-413c-9ccf-e8f35a13fc4a-maas-api-service-ca\") pod \"ca584214-2809-413c-9ccf-e8f35a13fc4a\" (UID: \"ca584214-2809-413c-9ccf-e8f35a13fc4a\") " Sep 08 19:04:09.983076 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:09.983052 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca584214-2809-413c-9ccf-e8f35a13fc4a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca584214-2809-413c-9ccf-e8f35a13fc4a" (UID: "ca584214-2809-413c-9ccf-e8f35a13fc4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:04:10.083629 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:10.083535 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca584214-2809-413c-9ccf-e8f35a13fc4a-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:04:10.766352 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:10.766327 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v" Sep 08 19:04:10.766501 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:10.766371 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v" event={"ID":"ca584214-2809-413c-9ccf-e8f35a13fc4a","Type":"ContainerDied","Data":"7f9a7ea11145e436e837d65fb6057151fbefa6507ab28422c3d9c5f2a653cbcb"} Sep 08 19:04:10.766501 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:10.766419 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f9a7ea11145e436e837d65fb6057151fbefa6507ab28422c3d9c5f2a653cbcb" Sep 08 19:04:16.738053 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:16.738022 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" Sep 08 19:04:16.738451 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:16.738095 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" Sep 08 19:04:17.587025 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:17.586995 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c6f2ed-29814900-n4vsx"] Sep 08 19:04:17.589512 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:17.589488 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c6f2ed-29814900-n4vsx"] Sep 08 19:04:17.608794 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:17.608764 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp"] Sep 08 19:04:17.609076 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:17.609047 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" podUID="4b317876-c273-434c-8ff1-9f3b6637a72d" containerName="payload-processing" containerID="cri-o://4190f0450f49a38030b54610edbde817330e527e50f74ee36033c87ef798926e" gracePeriod=30 Sep 08 19:04:17.774266 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:17.774232 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v"] Sep 08 19:04:17.777057 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:17.777025 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-c6f2ed-69f4v"] Sep 08 19:04:17.811550 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:17.811522 2553 generic.go:356] "Generic (PLEG): container finished" podID="4b317876-c273-434c-8ff1-9f3b6637a72d" containerID="4190f0450f49a38030b54610edbde817330e527e50f74ee36033c87ef798926e" exitCode=0 Sep 08 19:04:17.811660 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:17.811574 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" event={"ID":"4b317876-c273-434c-8ff1-9f3b6637a72d","Type":"ContainerDied","Data":"4190f0450f49a38030b54610edbde817330e527e50f74ee36033c87ef798926e"} Sep 08 19:04:17.903030 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:17.903003 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" Sep 08 19:04:17.960895 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:17.960862 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfd8p\" (UniqueName: \"kubernetes.io/projected/4b317876-c273-434c-8ff1-9f3b6637a72d-kube-api-access-cfd8p\") pod \"4b317876-c273-434c-8ff1-9f3b6637a72d\" (UID: \"4b317876-c273-434c-8ff1-9f3b6637a72d\") " Sep 08 19:04:17.961070 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:17.960923 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b317876-c273-434c-8ff1-9f3b6637a72d-plugins-config-volume\") pod \"4b317876-c273-434c-8ff1-9f3b6637a72d\" (UID: \"4b317876-c273-434c-8ff1-9f3b6637a72d\") " Sep 08 19:04:17.961289 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:17.961268 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b317876-c273-434c-8ff1-9f3b6637a72d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b317876-c273-434c-8ff1-9f3b6637a72d" (UID: "4b317876-c273-434c-8ff1-9f3b6637a72d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:04:17.962892 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:17.962872 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b317876-c273-434c-8ff1-9f3b6637a72d-kube-api-access-cfd8p" (OuterVolumeSpecName: "kube-api-access-cfd8p") pod "4b317876-c273-434c-8ff1-9f3b6637a72d" (UID: "4b317876-c273-434c-8ff1-9f3b6637a72d"). InnerVolumeSpecName "kube-api-access-cfd8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:04:18.062658 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:18.062628 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b317876-c273-434c-8ff1-9f3b6637a72d-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:04:18.062658 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:18.062655 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfd8p\" (UniqueName: \"kubernetes.io/projected/4b317876-c273-434c-8ff1-9f3b6637a72d-kube-api-access-cfd8p\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:04:18.745775 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:18.745742 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b3d6801-46b4-42fb-9887-eb8202523e54" path="/var/lib/kubelet/pods/8b3d6801-46b4-42fb-9887-eb8202523e54/volumes" Sep 08 19:04:18.746065 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:18.746053 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca584214-2809-413c-9ccf-e8f35a13fc4a" path="/var/lib/kubelet/pods/ca584214-2809-413c-9ccf-e8f35a13fc4a/volumes" Sep 08 19:04:18.818414 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:18.818386 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" Sep 08 19:04:18.818767 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:18.818411 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp" event={"ID":"4b317876-c273-434c-8ff1-9f3b6637a72d","Type":"ContainerDied","Data":"a50ffd45bd8c1d314e2a5d3df96614de7c1e1c5a6dabede2eb93813a8e5de22f"} Sep 08 19:04:18.818767 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:18.818483 2553 scope.go:117] "RemoveContainer" containerID="4190f0450f49a38030b54610edbde817330e527e50f74ee36033c87ef798926e" Sep 08 19:04:18.837320 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:18.837297 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp"] Sep 08 19:04:18.841190 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:18.841165 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c6f2ed-754cf46b8c-nnngp"] Sep 08 19:04:20.745224 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:20.745193 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b317876-c273-434c-8ff1-9f3b6637a72d" path="/var/lib/kubelet/pods/4b317876-c273-434c-8ff1-9f3b6637a72d/volumes" Sep 08 19:04:24.202614 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:24.202579 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8"] Sep 08 19:04:24.202997 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:24.202838 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" podUID="655bc981-3bba-4de0-ac3b-513b13b896ad" containerName="istio-proxy" containerID="cri-o://d4771622673c93f36419b1ac3a6b6e2a240d89c0c87b999fa0d03de4d381c0f0" gracePeriod=30 Sep 08 19:04:24.966654 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:24.966581 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 19:04:24.978056 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:24.978011 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 19:04:24.979933 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:24.979908 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 19:04:24.984249 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:24.984228 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 19:04:28.237310 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:28.237270 2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 19:04:28.237671 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:28.237335 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" podUID="655bc981-3bba-4de0-ac3b-513b13b896ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 19:04:29.460850 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.460823 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 19:04:29.569915 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.569815 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-token\") pod \"655bc981-3bba-4de0-ac3b-513b13b896ad\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " Sep 08 19:04:29.569915 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.569868 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655bc981-3bba-4de0-ac3b-513b13b896ad-istiod-ca-cert\") pod \"655bc981-3bba-4de0-ac3b-513b13b896ad\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " Sep 08 19:04:29.569915 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.569912 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwwtv\" (UniqueName: \"kubernetes.io/projected/655bc981-3bba-4de0-ac3b-513b13b896ad-kube-api-access-rwwtv\") pod \"655bc981-3bba-4de0-ac3b-513b13b896ad\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " Sep 08 19:04:29.570211 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.570006 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-podinfo\") pod \"655bc981-3bba-4de0-ac3b-513b13b896ad\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " Sep 08 19:04:29.570211 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.570076 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-workload-socket\") pod \"655bc981-3bba-4de0-ac3b-513b13b896ad\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " Sep 08 19:04:29.570211 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.570198 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-data\") pod \"655bc981-3bba-4de0-ac3b-513b13b896ad\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " Sep 08 19:04:29.570408 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.570243 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-workload-certs\") pod \"655bc981-3bba-4de0-ac3b-513b13b896ad\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " Sep 08 19:04:29.570408 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.570275 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "655bc981-3bba-4de0-ac3b-513b13b896ad" (UID: "655bc981-3bba-4de0-ac3b-513b13b896ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:04:29.570408 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.570288 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-envoy\") pod \"655bc981-3bba-4de0-ac3b-513b13b896ad\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " Sep 08 19:04:29.570408 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.570291 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655bc981-3bba-4de0-ac3b-513b13b896ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "655bc981-3bba-4de0-ac3b-513b13b896ad" (UID: "655bc981-3bba-4de0-ac3b-513b13b896ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:04:29.570408 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.570337 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-credential-socket\") pod \"655bc981-3bba-4de0-ac3b-513b13b896ad\" (UID: \"655bc981-3bba-4de0-ac3b-513b13b896ad\") " Sep 08 19:04:29.570658 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.570457 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "655bc981-3bba-4de0-ac3b-513b13b896ad" (UID: "655bc981-3bba-4de0-ac3b-513b13b896ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:04:29.570658 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.570472 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "655bc981-3bba-4de0-ac3b-513b13b896ad" (UID: "655bc981-3bba-4de0-ac3b-513b13b896ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:04:29.570767 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.570679 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "655bc981-3bba-4de0-ac3b-513b13b896ad" (UID: "655bc981-3bba-4de0-ac3b-513b13b896ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:04:29.570767 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.570758 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655bc981-3bba-4de0-ac3b-513b13b896ad-istiod-ca-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:04:29.570876 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.570779 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-workload-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:04:29.570876 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.570792 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-data\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:04:29.570876 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.570804 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-workload-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:04:29.572146 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.572110 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "655bc981-3bba-4de0-ac3b-513b13b896ad" (UID: "655bc981-3bba-4de0-ac3b-513b13b896ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 19:04:29.572287 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.572215 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "655bc981-3bba-4de0-ac3b-513b13b896ad" (UID: "655bc981-3bba-4de0-ac3b-513b13b896ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:04:29.572287 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.572243 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655bc981-3bba-4de0-ac3b-513b13b896ad-kube-api-access-rwwtv" (OuterVolumeSpecName: "kube-api-access-rwwtv") pod "655bc981-3bba-4de0-ac3b-513b13b896ad" (UID: "655bc981-3bba-4de0-ac3b-513b13b896ad"). InnerVolumeSpecName "kube-api-access-rwwtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:04:29.572399 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.572321 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "655bc981-3bba-4de0-ac3b-513b13b896ad" (UID: "655bc981-3bba-4de0-ac3b-513b13b896ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 19:04:29.671940 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.671913 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwwtv\" (UniqueName: \"kubernetes.io/projected/655bc981-3bba-4de0-ac3b-513b13b896ad-kube-api-access-rwwtv\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:04:29.671940 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.671941 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-podinfo\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:04:29.672085 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.671954 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-envoy\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:04:29.672085 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.671966 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655bc981-3bba-4de0-ac3b-513b13b896ad-credential-socket\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:04:29.672085 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.671980 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655bc981-3bba-4de0-ac3b-513b13b896ad-istio-token\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:04:29.882955 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.882928 2553 generic.go:356] "Generic (PLEG): container finished" podID="655bc981-3bba-4de0-ac3b-513b13b896ad" containerID="d4771622673c93f36419b1ac3a6b6e2a240d89c0c87b999fa0d03de4d381c0f0" exitCode=0 Sep 08 19:04:29.883129 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.882971 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" Sep 08 19:04:29.883129 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.882989 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" event={"ID":"655bc981-3bba-4de0-ac3b-513b13b896ad","Type":"ContainerDied","Data":"d4771622673c93f36419b1ac3a6b6e2a240d89c0c87b999fa0d03de4d381c0f0"} Sep 08 19:04:29.883129 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.883041 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8" event={"ID":"655bc981-3bba-4de0-ac3b-513b13b896ad","Type":"ContainerDied","Data":"e390257c9ee9c8c905802faf08347289a744ccd5bd6a439fa0e8a79d3caf60a1"} Sep 08 19:04:29.883129 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.883068 2553 scope.go:117] "RemoveContainer" containerID="d4771622673c93f36419b1ac3a6b6e2a240d89c0c87b999fa0d03de4d381c0f0" Sep 08 19:04:29.894316 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.894298 2553 scope.go:117] "RemoveContainer" containerID="d74d9c437ee799093b58a7505ba03ee96ef3f8a82fbdafe06f7420de73e2c42f" Sep 08 19:04:29.902482 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.902461 2553 scope.go:117] "RemoveContainer" containerID="d4771622673c93f36419b1ac3a6b6e2a240d89c0c87b999fa0d03de4d381c0f0" Sep 08 19:04:29.902748 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:04:29.902725 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4771622673c93f36419b1ac3a6b6e2a240d89c0c87b999fa0d03de4d381c0f0\": container with ID starting with d4771622673c93f36419b1ac3a6b6e2a240d89c0c87b999fa0d03de4d381c0f0 not found: ID does not exist" containerID="d4771622673c93f36419b1ac3a6b6e2a240d89c0c87b999fa0d03de4d381c0f0" Sep 08 19:04:29.902805 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.902756 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4771622673c93f36419b1ac3a6b6e2a240d89c0c87b999fa0d03de4d381c0f0"} err="failed to get container status \"d4771622673c93f36419b1ac3a6b6e2a240d89c0c87b999fa0d03de4d381c0f0\": rpc error: code = NotFound desc = could not find container \"d4771622673c93f36419b1ac3a6b6e2a240d89c0c87b999fa0d03de4d381c0f0\": container with ID starting with d4771622673c93f36419b1ac3a6b6e2a240d89c0c87b999fa0d03de4d381c0f0 not found: ID does not exist" Sep 08 19:04:29.902805 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.902773 2553 scope.go:117] "RemoveContainer" containerID="d74d9c437ee799093b58a7505ba03ee96ef3f8a82fbdafe06f7420de73e2c42f" Sep 08 19:04:29.903013 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:04:29.902997 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d74d9c437ee799093b58a7505ba03ee96ef3f8a82fbdafe06f7420de73e2c42f\": container with ID starting with d74d9c437ee799093b58a7505ba03ee96ef3f8a82fbdafe06f7420de73e2c42f not found: ID does not exist" containerID="d74d9c437ee799093b58a7505ba03ee96ef3f8a82fbdafe06f7420de73e2c42f" Sep 08 19:04:29.903063 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.903019 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d74d9c437ee799093b58a7505ba03ee96ef3f8a82fbdafe06f7420de73e2c42f"} err="failed to get container status \"d74d9c437ee799093b58a7505ba03ee96ef3f8a82fbdafe06f7420de73e2c42f\": rpc error: code = NotFound desc = could not find container \"d74d9c437ee799093b58a7505ba03ee96ef3f8a82fbdafe06f7420de73e2c42f\": container with ID starting with d74d9c437ee799093b58a7505ba03ee96ef3f8a82fbdafe06f7420de73e2c42f not found: ID does not exist" Sep 08 19:04:29.930531 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.930503 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8"] Sep 08 19:04:29.933223 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:29.933197 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c6f2ed-openshift-default-6c65f8b9fb-jnrf8"] Sep 08 19:04:30.745161 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:30.745128 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="655bc981-3bba-4de0-ac3b-513b13b896ad" path="/var/lib/kubelet/pods/655bc981-3bba-4de0-ac3b-513b13b896ad/volumes" Sep 08 19:04:33.525654 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.525618 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h"] Sep 08 19:04:33.526057 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.526026 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b317876-c273-434c-8ff1-9f3b6637a72d" containerName="payload-processing" Sep 08 19:04:33.526057 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.526038 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b317876-c273-434c-8ff1-9f3b6637a72d" containerName="payload-processing" Sep 08 19:04:33.526057 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.526051 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="655bc981-3bba-4de0-ac3b-513b13b896ad" containerName="istio-proxy" Sep 08 19:04:33.526159 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.526060 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="655bc981-3bba-4de0-ac3b-513b13b896ad" containerName="istio-proxy" Sep 08 19:04:33.526159 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.526072 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca584214-2809-413c-9ccf-e8f35a13fc4a" containerName="revoke-keys" Sep 08 19:04:33.526159 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.526079 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca584214-2809-413c-9ccf-e8f35a13fc4a" containerName="revoke-keys" Sep 08 19:04:33.526159 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.526088 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="655bc981-3bba-4de0-ac3b-513b13b896ad" containerName="istio-proxy" Sep 08 19:04:33.526159 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.526094 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="655bc981-3bba-4de0-ac3b-513b13b896ad" containerName="istio-proxy" Sep 08 19:04:33.526159 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.526156 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="655bc981-3bba-4de0-ac3b-513b13b896ad" containerName="istio-proxy" Sep 08 19:04:33.526341 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.526165 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="655bc981-3bba-4de0-ac3b-513b13b896ad" containerName="istio-proxy" Sep 08 19:04:33.526341 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.526174 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b317876-c273-434c-8ff1-9f3b6637a72d" containerName="payload-processing" Sep 08 19:04:33.526341 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.526180 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca584214-2809-413c-9ccf-e8f35a13fc4a" containerName="revoke-keys" Sep 08 19:04:33.530507 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.530491 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" Sep 08 19:04:33.533496 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.533475 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1a27c5-dockercfg-96b9t\"" Sep 08 19:04:33.533918 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.533902 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1a27c5\"" Sep 08 19:04:33.539007 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.538985 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h"] Sep 08 19:04:33.710738 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.710700 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwk92\" (UniqueName: \"kubernetes.io/projected/5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd-kube-api-access-pwk92\") pod \"payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h\" (UID: \"5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" Sep 08 19:04:33.710915 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.710796 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h\" (UID: \"5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" Sep 08 19:04:33.811333 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.811234 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwk92\" (UniqueName: \"kubernetes.io/projected/5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd-kube-api-access-pwk92\") pod \"payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h\" (UID: \"5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" Sep 08 19:04:33.811333 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.811318 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h\" (UID: \"5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" Sep 08 19:04:33.812063 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.812041 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h\" (UID: \"5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" Sep 08 19:04:33.820263 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.820236 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwk92\" (UniqueName: \"kubernetes.io/projected/5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd-kube-api-access-pwk92\") pod \"payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h\" (UID: \"5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" Sep 08 19:04:33.843005 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:33.842965 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" Sep 08 19:04:34.119507 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:34.119478 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h"] Sep 08 19:04:34.120753 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:04:34.120729 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebb7468_d6d9_4c86_b8ca_4e6f7c2945fd.slice/crio-57a8ddd710a18e4c9ee4dd39ceaed47d35abe971f8598a0b6453608161aa88ad WatchSource:0}: Error finding container 57a8ddd710a18e4c9ee4dd39ceaed47d35abe971f8598a0b6453608161aa88ad: Status 404 returned error can't find the container with id 57a8ddd710a18e4c9ee4dd39ceaed47d35abe971f8598a0b6453608161aa88ad Sep 08 19:04:34.917975 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:34.917943 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" event={"ID":"5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd","Type":"ContainerStarted","Data":"f7840251baf2ad73a4f45c3d3c08632e2daf2a38fe79f079bd578353c33289bf"} Sep 08 19:04:34.918386 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:34.918000 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" event={"ID":"5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd","Type":"ContainerStarted","Data":"57a8ddd710a18e4c9ee4dd39ceaed47d35abe971f8598a0b6453608161aa88ad"} Sep 08 19:04:34.918386 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:34.918043 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" Sep 08 19:04:34.934183 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:34.934137 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" podStartSLOduration=1.934123541 podStartE2EDuration="1.934123541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:04:34.933181077 +0000 UTC m=+1810.639069310" watchObservedRunningTime="2026-09-08 19:04:34.934123541 +0000 UTC m=+1810.640011815" Sep 08 19:04:45.925667 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:04:45.925632 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" Sep 08 19:05:42.701823 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:42.701788 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6_35bd4061-c9b0-4fd4-a1cf-ec835b614d44/payload-processing/0.log" Sep 08 19:05:47.966675 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:47.966597 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw"] Sep 08 19:05:47.970242 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:47.970225 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" Sep 08 19:05:47.973291 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:47.973267 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-3a21b6\"" Sep 08 19:05:47.973583 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:47.973555 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-3a21b6-dockercfg-2plqb\"" Sep 08 19:05:47.980877 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:47.980852 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw"] Sep 08 19:05:48.015485 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.015454 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx"] Sep 08 19:05:48.018976 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.018954 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" Sep 08 19:05:48.031852 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.031826 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx"] Sep 08 19:05:48.066503 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.066472 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7xk2\" (UniqueName: \"kubernetes.io/projected/854380ae-405a-4b94-b67c-de703b5f9837-kube-api-access-c7xk2\") pod \"payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw\" (UID: \"854380ae-405a-4b94-b67c-de703b5f9837\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" Sep 08 19:05:48.066670 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.066521 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854380ae-405a-4b94-b67c-de703b5f9837-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw\" (UID: \"854380ae-405a-4b94-b67c-de703b5f9837\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" Sep 08 19:05:48.066670 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.066592 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zstcn\" (UniqueName: \"kubernetes.io/projected/f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf-kube-api-access-zstcn\") pod \"payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx\" (UID: \"f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" Sep 08 19:05:48.066670 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.066625 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx\" (UID: \"f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" Sep 08 19:05:48.167665 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.167631 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zstcn\" (UniqueName: \"kubernetes.io/projected/f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf-kube-api-access-zstcn\") pod \"payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx\" (UID: \"f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" Sep 08 19:05:48.167831 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.167675 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx\" (UID: \"f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" Sep 08 19:05:48.167831 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.167764 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7xk2\" (UniqueName: \"kubernetes.io/projected/854380ae-405a-4b94-b67c-de703b5f9837-kube-api-access-c7xk2\") pod \"payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw\" (UID: \"854380ae-405a-4b94-b67c-de703b5f9837\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" Sep 08 19:05:48.167831 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.167795 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854380ae-405a-4b94-b67c-de703b5f9837-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw\" (UID: \"854380ae-405a-4b94-b67c-de703b5f9837\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" Sep 08 19:05:48.168704 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.168680 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854380ae-405a-4b94-b67c-de703b5f9837-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw\" (UID: \"854380ae-405a-4b94-b67c-de703b5f9837\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" Sep 08 19:05:48.168762 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.168680 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx\" (UID: \"f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" Sep 08 19:05:48.175828 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.175802 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zstcn\" (UniqueName: \"kubernetes.io/projected/f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf-kube-api-access-zstcn\") pod \"payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx\" (UID: \"f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" Sep 08 19:05:48.176485 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.176453 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7xk2\" (UniqueName: \"kubernetes.io/projected/854380ae-405a-4b94-b67c-de703b5f9837-kube-api-access-c7xk2\") pod \"payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw\" (UID: \"854380ae-405a-4b94-b67c-de703b5f9837\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" Sep 08 19:05:48.282266 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.282191 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" Sep 08 19:05:48.330835 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.330307 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" Sep 08 19:05:48.554720 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.554686 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw"] Sep 08 19:05:48.555578 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:05:48.555552 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854380ae_405a_4b94_b67c_de703b5f9837.slice/crio-dc418f079064a99e064ef133ad3b5335ffaea0e2084a2fa70f9869f531660f0c WatchSource:0}: Error finding container dc418f079064a99e064ef133ad3b5335ffaea0e2084a2fa70f9869f531660f0c: Status 404 returned error can't find the container with id dc418f079064a99e064ef133ad3b5335ffaea0e2084a2fa70f9869f531660f0c Sep 08 19:05:48.613799 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:48.613772 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx"] Sep 08 19:05:48.618483 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:05:48.618457 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf841f0bb_59a3_4e3e_bb9c_d5fe77953aaf.slice/crio-47cad98070bbbe2ed1d0e0475ffe61781f22ee1dacc260321fa5705f2230ca7e WatchSource:0}: Error finding container 47cad98070bbbe2ed1d0e0475ffe61781f22ee1dacc260321fa5705f2230ca7e: Status 404 returned error can't find the container with id 47cad98070bbbe2ed1d0e0475ffe61781f22ee1dacc260321fa5705f2230ca7e Sep 08 19:05:49.359854 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:49.359812 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" event={"ID":"f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf","Type":"ContainerStarted","Data":"a548bd82798907cedf9ef0a06336c3bea36b73098610e540971c442f39d28b5a"} Sep 08 19:05:49.360324 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:49.359877 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" event={"ID":"f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf","Type":"ContainerStarted","Data":"47cad98070bbbe2ed1d0e0475ffe61781f22ee1dacc260321fa5705f2230ca7e"} Sep 08 19:05:49.360324 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:49.359906 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" Sep 08 19:05:49.361549 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:49.361515 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" event={"ID":"854380ae-405a-4b94-b67c-de703b5f9837","Type":"ContainerStarted","Data":"edbc66a0d1d6d3223558e75b642d929c853714d0948e003e89ee9d0cb85e143b"} Sep 08 19:05:49.361665 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:49.361568 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" event={"ID":"854380ae-405a-4b94-b67c-de703b5f9837","Type":"ContainerStarted","Data":"dc418f079064a99e064ef133ad3b5335ffaea0e2084a2fa70f9869f531660f0c"} Sep 08 19:05:49.361665 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:49.361593 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" Sep 08 19:05:49.377162 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:49.377119 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" podStartSLOduration=2.377105887 podStartE2EDuration="2.377105887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:05:49.3748151 +0000 UTC m=+1885.080703344" watchObservedRunningTime="2026-09-08 19:05:49.377105887 +0000 UTC m=+1885.082994118" Sep 08 19:05:49.389674 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:05:49.389631 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" podStartSLOduration=2.389617309 podStartE2EDuration="2.389617309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:05:49.388661826 +0000 UTC m=+1885.094550059" watchObservedRunningTime="2026-09-08 19:05:49.389617309 +0000 UTC m=+1885.095505541" Sep 08 19:06:00.368902 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:00.368865 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" Sep 08 19:06:00.369356 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:00.369043 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" Sep 08 19:06:11.958621 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:11.958582 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk"] Sep 08 19:06:11.962449 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:11.962425 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk" Sep 08 19:06:11.966028 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:11.966005 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 19:06:11.966909 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:11.966861 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 19:06:11.972530 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:11.972505 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk"] Sep 08 19:06:12.090900 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:12.090860 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d2cfea-73d8-4ede-a3e2-15ee2ea95c85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk\" (UID: \"81d2cfea-73d8-4ede-a3e2-15ee2ea95c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk" Sep 08 19:06:12.192311 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:12.192275 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d2cfea-73d8-4ede-a3e2-15ee2ea95c85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk\" (UID: \"81d2cfea-73d8-4ede-a3e2-15ee2ea95c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk" Sep 08 19:06:12.192938 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:12.192912 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d2cfea-73d8-4ede-a3e2-15ee2ea95c85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk\" (UID: \"81d2cfea-73d8-4ede-a3e2-15ee2ea95c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk" Sep 08 19:06:12.275718 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:12.275643 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk" Sep 08 19:06:12.548834 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:12.548806 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk"] Sep 08 19:06:12.551085 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:06:12.551043 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d2cfea_73d8_4ede_a3e2_15ee2ea95c85.slice/crio-069bc72781884af26d45d7378251425feee7a1ba0745cf4dfea16f0b31c2b81b WatchSource:0}: Error finding container 069bc72781884af26d45d7378251425feee7a1ba0745cf4dfea16f0b31c2b81b: Status 404 returned error can't find the container with id 069bc72781884af26d45d7378251425feee7a1ba0745cf4dfea16f0b31c2b81b Sep 08 19:06:13.502674 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:13.502580 2553 generic.go:356] "Generic (PLEG): container finished" podID="81d2cfea-73d8-4ede-a3e2-15ee2ea95c85" containerID="a5108bb8f6b6642b8bc193085a12e06131b54fd682772fa5ba56e148f4ae135b" exitCode=0 Sep 08 19:06:13.503068 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:13.502665 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk" event={"ID":"81d2cfea-73d8-4ede-a3e2-15ee2ea95c85","Type":"ContainerDied","Data":"a5108bb8f6b6642b8bc193085a12e06131b54fd682772fa5ba56e148f4ae135b"} Sep 08 19:06:13.503068 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:13.502741 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk" event={"ID":"81d2cfea-73d8-4ede-a3e2-15ee2ea95c85","Type":"ContainerStarted","Data":"069bc72781884af26d45d7378251425feee7a1ba0745cf4dfea16f0b31c2b81b"} Sep 08 19:06:14.644505 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:14.644479 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk" Sep 08 19:06:14.711249 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:14.711215 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d2cfea-73d8-4ede-a3e2-15ee2ea95c85-maas-api-service-ca\") pod \"81d2cfea-73d8-4ede-a3e2-15ee2ea95c85\" (UID: \"81d2cfea-73d8-4ede-a3e2-15ee2ea95c85\") " Sep 08 19:06:14.711556 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:14.711533 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d2cfea-73d8-4ede-a3e2-15ee2ea95c85-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "81d2cfea-73d8-4ede-a3e2-15ee2ea95c85" (UID: "81d2cfea-73d8-4ede-a3e2-15ee2ea95c85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:06:14.812678 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:14.812604 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d2cfea-73d8-4ede-a3e2-15ee2ea95c85-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:06:15.517242 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:15.517212 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk" Sep 08 19:06:15.517242 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:15.517235 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk" event={"ID":"81d2cfea-73d8-4ede-a3e2-15ee2ea95c85","Type":"ContainerDied","Data":"069bc72781884af26d45d7378251425feee7a1ba0745cf4dfea16f0b31c2b81b"} Sep 08 19:06:15.517471 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:15.517282 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="069bc72781884af26d45d7378251425feee7a1ba0745cf4dfea16f0b31c2b81b" Sep 08 19:06:17.945989 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:17.945943 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx"] Sep 08 19:06:17.946515 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:17.946197 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" podUID="f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf" containerName="payload-processing" containerID="cri-o://a548bd82798907cedf9ef0a06336c3bea36b73098610e540971c442f39d28b5a" gracePeriod=30 Sep 08 19:06:17.951623 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:17.951283 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw"] Sep 08 19:06:17.951623 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:17.951506 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" podUID="854380ae-405a-4b94-b67c-de703b5f9837" containerName="payload-pre-processing" containerID="cri-o://edbc66a0d1d6d3223558e75b642d929c853714d0948e003e89ee9d0cb85e143b" gracePeriod=30 Sep 08 19:06:18.104864 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.104828 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk"] Sep 08 19:06:18.111754 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.111693 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3a21b6-klzgk"] Sep 08 19:06:18.219927 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.219902 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" Sep 08 19:06:18.243258 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.243222 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zstcn\" (UniqueName: \"kubernetes.io/projected/f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf-kube-api-access-zstcn\") pod \"f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf\" (UID: \"f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf\") " Sep 08 19:06:18.243812 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.243790 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf-plugins-config-volume\") pod \"f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf\" (UID: \"f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf\") " Sep 08 19:06:18.244212 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.244184 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf" (UID: "f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:06:18.244671 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.244520 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:06:18.246010 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.245938 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf-kube-api-access-zstcn" (OuterVolumeSpecName: "kube-api-access-zstcn") pod "f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf" (UID: "f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf"). InnerVolumeSpecName "kube-api-access-zstcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:06:18.256456 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.256411 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" Sep 08 19:06:18.344839 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.344811 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854380ae-405a-4b94-b67c-de703b5f9837-plugins-config-volume\") pod \"854380ae-405a-4b94-b67c-de703b5f9837\" (UID: \"854380ae-405a-4b94-b67c-de703b5f9837\") " Sep 08 19:06:18.345006 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.344897 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7xk2\" (UniqueName: \"kubernetes.io/projected/854380ae-405a-4b94-b67c-de703b5f9837-kube-api-access-c7xk2\") pod \"854380ae-405a-4b94-b67c-de703b5f9837\" (UID: \"854380ae-405a-4b94-b67c-de703b5f9837\") " Sep 08 19:06:18.345063 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.345052 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zstcn\" (UniqueName: \"kubernetes.io/projected/f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf-kube-api-access-zstcn\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:06:18.345230 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.345204 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/854380ae-405a-4b94-b67c-de703b5f9837-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "854380ae-405a-4b94-b67c-de703b5f9837" (UID: "854380ae-405a-4b94-b67c-de703b5f9837"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:06:18.346955 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.346937 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854380ae-405a-4b94-b67c-de703b5f9837-kube-api-access-c7xk2" (OuterVolumeSpecName: "kube-api-access-c7xk2") pod "854380ae-405a-4b94-b67c-de703b5f9837" (UID: "854380ae-405a-4b94-b67c-de703b5f9837"). InnerVolumeSpecName "kube-api-access-c7xk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:06:18.446239 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.446208 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/854380ae-405a-4b94-b67c-de703b5f9837-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:06:18.446239 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.446238 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7xk2\" (UniqueName: \"kubernetes.io/projected/854380ae-405a-4b94-b67c-de703b5f9837-kube-api-access-c7xk2\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:06:18.537090 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.537019 2553 generic.go:356] "Generic (PLEG): container finished" podID="854380ae-405a-4b94-b67c-de703b5f9837" containerID="edbc66a0d1d6d3223558e75b642d929c853714d0948e003e89ee9d0cb85e143b" exitCode=0 Sep 08 19:06:18.537090 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.537063 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" Sep 08 19:06:18.537230 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.537104 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" event={"ID":"854380ae-405a-4b94-b67c-de703b5f9837","Type":"ContainerDied","Data":"edbc66a0d1d6d3223558e75b642d929c853714d0948e003e89ee9d0cb85e143b"} Sep 08 19:06:18.537230 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.537185 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw" event={"ID":"854380ae-405a-4b94-b67c-de703b5f9837","Type":"ContainerDied","Data":"dc418f079064a99e064ef133ad3b5335ffaea0e2084a2fa70f9869f531660f0c"} Sep 08 19:06:18.537230 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.537217 2553 scope.go:117] "RemoveContainer" containerID="edbc66a0d1d6d3223558e75b642d929c853714d0948e003e89ee9d0cb85e143b" Sep 08 19:06:18.538608 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.538575 2553 generic.go:356] "Generic (PLEG): container finished" podID="f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf" containerID="a548bd82798907cedf9ef0a06336c3bea36b73098610e540971c442f39d28b5a" exitCode=0 Sep 08 19:06:18.538715 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.538623 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" Sep 08 19:06:18.538715 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.538631 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" event={"ID":"f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf","Type":"ContainerDied","Data":"a548bd82798907cedf9ef0a06336c3bea36b73098610e540971c442f39d28b5a"} Sep 08 19:06:18.538715 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.538708 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx" event={"ID":"f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf","Type":"ContainerDied","Data":"47cad98070bbbe2ed1d0e0475ffe61781f22ee1dacc260321fa5705f2230ca7e"} Sep 08 19:06:18.548509 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.548490 2553 scope.go:117] "RemoveContainer" containerID="edbc66a0d1d6d3223558e75b642d929c853714d0948e003e89ee9d0cb85e143b" Sep 08 19:06:18.548804 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:06:18.548781 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edbc66a0d1d6d3223558e75b642d929c853714d0948e003e89ee9d0cb85e143b\": container with ID starting with edbc66a0d1d6d3223558e75b642d929c853714d0948e003e89ee9d0cb85e143b not found: ID does not exist" containerID="edbc66a0d1d6d3223558e75b642d929c853714d0948e003e89ee9d0cb85e143b" Sep 08 19:06:18.548855 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.548815 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edbc66a0d1d6d3223558e75b642d929c853714d0948e003e89ee9d0cb85e143b"} err="failed to get container status \"edbc66a0d1d6d3223558e75b642d929c853714d0948e003e89ee9d0cb85e143b\": rpc error: code = NotFound desc = could not find container \"edbc66a0d1d6d3223558e75b642d929c853714d0948e003e89ee9d0cb85e143b\": container with ID starting with edbc66a0d1d6d3223558e75b642d929c853714d0948e003e89ee9d0cb85e143b not found: ID does not exist" Sep 08 19:06:18.548855 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.548838 2553 scope.go:117] "RemoveContainer" containerID="a548bd82798907cedf9ef0a06336c3bea36b73098610e540971c442f39d28b5a" Sep 08 19:06:18.557206 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.557184 2553 scope.go:117] "RemoveContainer" containerID="a548bd82798907cedf9ef0a06336c3bea36b73098610e540971c442f39d28b5a" Sep 08 19:06:18.557449 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:06:18.557433 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a548bd82798907cedf9ef0a06336c3bea36b73098610e540971c442f39d28b5a\": container with ID starting with a548bd82798907cedf9ef0a06336c3bea36b73098610e540971c442f39d28b5a not found: ID does not exist" containerID="a548bd82798907cedf9ef0a06336c3bea36b73098610e540971c442f39d28b5a" Sep 08 19:06:18.557508 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.557455 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a548bd82798907cedf9ef0a06336c3bea36b73098610e540971c442f39d28b5a"} err="failed to get container status \"a548bd82798907cedf9ef0a06336c3bea36b73098610e540971c442f39d28b5a\": rpc error: code = NotFound desc = could not find container \"a548bd82798907cedf9ef0a06336c3bea36b73098610e540971c442f39d28b5a\": container with ID starting with a548bd82798907cedf9ef0a06336c3bea36b73098610e540971c442f39d28b5a not found: ID does not exist" Sep 08 19:06:18.564427 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.564403 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx"] Sep 08 19:06:18.568171 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.568144 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3a21b6-588b5656df-4g4lx"] Sep 08 19:06:18.578127 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.578102 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw"] Sep 08 19:06:18.581876 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.581856 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3a21b6-f5455b8cd-6wmlw"] Sep 08 19:06:18.745340 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.745307 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81d2cfea-73d8-4ede-a3e2-15ee2ea95c85" path="/var/lib/kubelet/pods/81d2cfea-73d8-4ede-a3e2-15ee2ea95c85/volumes" Sep 08 19:06:18.745713 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.745697 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="854380ae-405a-4b94-b67c-de703b5f9837" path="/var/lib/kubelet/pods/854380ae-405a-4b94-b67c-de703b5f9837/volumes" Sep 08 19:06:18.746036 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:18.746024 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf" path="/var/lib/kubelet/pods/f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf/volumes" Sep 08 19:06:25.139421 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:25.139388 2553 scope.go:117] "RemoveContainer" containerID="d7035e3747c0acd8fbcff0065e454feb59d0146853a4182decc4ab10c6ca3fa4" Sep 08 19:06:25.148189 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:25.148167 2553 scope.go:117] "RemoveContainer" containerID="00c3dc8c58ae3e9bc2f82dac9d612072f3dcb207b8ba79d82532410ca8318dcf" Sep 08 19:06:28.846435 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:28.846343 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v"] Sep 08 19:06:28.846803 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:28.846735 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81d2cfea-73d8-4ede-a3e2-15ee2ea95c85" containerName="revoke-keys" Sep 08 19:06:28.846803 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:28.846748 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="81d2cfea-73d8-4ede-a3e2-15ee2ea95c85" containerName="revoke-keys" Sep 08 19:06:28.846803 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:28.846767 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf" containerName="payload-processing" Sep 08 19:06:28.846803 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:28.846776 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf" containerName="payload-processing" Sep 08 19:06:28.846803 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:28.846802 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="854380ae-405a-4b94-b67c-de703b5f9837" containerName="payload-pre-processing" Sep 08 19:06:28.846979 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:28.846808 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="854380ae-405a-4b94-b67c-de703b5f9837" containerName="payload-pre-processing" Sep 08 19:06:28.846979 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:28.846874 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="854380ae-405a-4b94-b67c-de703b5f9837" containerName="payload-pre-processing" Sep 08 19:06:28.846979 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:28.846885 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f841f0bb-59a3-4e3e-bb9c-d5fe77953aaf" containerName="payload-processing" Sep 08 19:06:28.846979 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:28.846892 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="81d2cfea-73d8-4ede-a3e2-15ee2ea95c85" containerName="revoke-keys" Sep 08 19:06:28.849947 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:28.849929 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v" Sep 08 19:06:28.852516 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:28.852491 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 19:06:28.852652 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:28.852516 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 19:06:28.858149 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:28.858117 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v"] Sep 08 19:06:28.946402 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:28.946348 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4c6e3a-63a4-45b2-b085-24310123bd1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v\" (UID: \"fa4c6e3a-63a4-45b2-b085-24310123bd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v" Sep 08 19:06:29.047929 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:29.047892 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4c6e3a-63a4-45b2-b085-24310123bd1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v\" (UID: \"fa4c6e3a-63a4-45b2-b085-24310123bd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v" Sep 08 19:06:29.048516 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:29.048498 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4c6e3a-63a4-45b2-b085-24310123bd1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v\" (UID: \"fa4c6e3a-63a4-45b2-b085-24310123bd1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v" Sep 08 19:06:29.162131 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:29.162103 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v" Sep 08 19:06:29.433534 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:29.433489 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v"] Sep 08 19:06:29.434528 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:06:29.434505 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4c6e3a_63a4_45b2_b085_24310123bd1e.slice/crio-f502ed6615c17a30135fba4977f6ea34553318f26262594eff11b36629aa88a0 WatchSource:0}: Error finding container f502ed6615c17a30135fba4977f6ea34553318f26262594eff11b36629aa88a0: Status 404 returned error can't find the container with id f502ed6615c17a30135fba4977f6ea34553318f26262594eff11b36629aa88a0 Sep 08 19:06:29.607545 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:29.607509 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v" event={"ID":"fa4c6e3a-63a4-45b2-b085-24310123bd1e","Type":"ContainerStarted","Data":"f502ed6615c17a30135fba4977f6ea34553318f26262594eff11b36629aa88a0"} Sep 08 19:06:30.616074 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:30.615973 2553 generic.go:356] "Generic (PLEG): container finished" podID="fa4c6e3a-63a4-45b2-b085-24310123bd1e" containerID="2a0404b902d4d7c46a3ce9593ca77789a3e35e527df813b9c992cc3ca01f6822" exitCode=0 Sep 08 19:06:30.616527 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:30.616063 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v" event={"ID":"fa4c6e3a-63a4-45b2-b085-24310123bd1e","Type":"ContainerDied","Data":"2a0404b902d4d7c46a3ce9593ca77789a3e35e527df813b9c992cc3ca01f6822"} Sep 08 19:06:31.753379 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:31.753344 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v" Sep 08 19:06:31.871166 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:31.871131 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4c6e3a-63a4-45b2-b085-24310123bd1e-maas-api-service-ca\") pod \"fa4c6e3a-63a4-45b2-b085-24310123bd1e\" (UID: \"fa4c6e3a-63a4-45b2-b085-24310123bd1e\") " Sep 08 19:06:31.871442 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:31.871418 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4c6e3a-63a4-45b2-b085-24310123bd1e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa4c6e3a-63a4-45b2-b085-24310123bd1e" (UID: "fa4c6e3a-63a4-45b2-b085-24310123bd1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:06:31.972817 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:31.972732 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa4c6e3a-63a4-45b2-b085-24310123bd1e-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:06:32.630243 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:32.630204 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v" event={"ID":"fa4c6e3a-63a4-45b2-b085-24310123bd1e","Type":"ContainerDied","Data":"f502ed6615c17a30135fba4977f6ea34553318f26262594eff11b36629aa88a0"} Sep 08 19:06:32.630499 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:32.630272 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f502ed6615c17a30135fba4977f6ea34553318f26262594eff11b36629aa88a0" Sep 08 19:06:32.630499 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:32.630215 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v" Sep 08 19:06:44.373487 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.373450 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6"] Sep 08 19:06:44.374016 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.373714 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" podUID="35bd4061-c9b0-4fd4-a1cf-ec835b614d44" containerName="payload-processing" containerID="cri-o://b4c8f83bad49085565f0ffd7f6f1fdbf515316ec7f05a70eb2eb74e531867fa9" gracePeriod=30 Sep 08 19:06:44.383046 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.383021 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr"] Sep 08 19:06:44.383269 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.383232 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" podUID="25813327-9a1c-4b26-8417-58aeba664091" containerName="payload-pre-processing" containerID="cri-o://91e008a429b9e89965376f4032927df6ffc5402b26838c4454a2730a7fd6343b" gracePeriod=30 Sep 08 19:06:44.535473 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.535430 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v"] Sep 08 19:06:44.538185 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.538159 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e77525-zjm6v"] Sep 08 19:06:44.656779 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.656758 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" Sep 08 19:06:44.670313 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.670290 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" Sep 08 19:06:44.700550 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.700520 2553 generic.go:356] "Generic (PLEG): container finished" podID="25813327-9a1c-4b26-8417-58aeba664091" containerID="91e008a429b9e89965376f4032927df6ffc5402b26838c4454a2730a7fd6343b" exitCode=0 Sep 08 19:06:44.700717 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.700559 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" Sep 08 19:06:44.700717 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.700594 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" event={"ID":"25813327-9a1c-4b26-8417-58aeba664091","Type":"ContainerDied","Data":"91e008a429b9e89965376f4032927df6ffc5402b26838c4454a2730a7fd6343b"} Sep 08 19:06:44.700717 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.700645 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr" event={"ID":"25813327-9a1c-4b26-8417-58aeba664091","Type":"ContainerDied","Data":"3d044997f4ad033bd26613fe84ba45d3f397dba64909227ff5ab78ba6db87830"} Sep 08 19:06:44.700717 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.700676 2553 scope.go:117] "RemoveContainer" containerID="91e008a429b9e89965376f4032927df6ffc5402b26838c4454a2730a7fd6343b" Sep 08 19:06:44.702085 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.702060 2553 generic.go:356] "Generic (PLEG): container finished" podID="35bd4061-c9b0-4fd4-a1cf-ec835b614d44" containerID="b4c8f83bad49085565f0ffd7f6f1fdbf515316ec7f05a70eb2eb74e531867fa9" exitCode=0 Sep 08 19:06:44.702187 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.702099 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" Sep 08 19:06:44.702187 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.702138 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" event={"ID":"35bd4061-c9b0-4fd4-a1cf-ec835b614d44","Type":"ContainerDied","Data":"b4c8f83bad49085565f0ffd7f6f1fdbf515316ec7f05a70eb2eb74e531867fa9"} Sep 08 19:06:44.702187 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.702180 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6" event={"ID":"35bd4061-c9b0-4fd4-a1cf-ec835b614d44","Type":"ContainerDied","Data":"f42580e8d17e2e2ace321a51b259a7d70cffe236e85521be1c41be7851cc3954"} Sep 08 19:06:44.712517 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.712489 2553 scope.go:117] "RemoveContainer" containerID="91e008a429b9e89965376f4032927df6ffc5402b26838c4454a2730a7fd6343b" Sep 08 19:06:44.712787 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:06:44.712770 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e008a429b9e89965376f4032927df6ffc5402b26838c4454a2730a7fd6343b\": container with ID starting with 91e008a429b9e89965376f4032927df6ffc5402b26838c4454a2730a7fd6343b not found: ID does not exist" containerID="91e008a429b9e89965376f4032927df6ffc5402b26838c4454a2730a7fd6343b" Sep 08 19:06:44.712834 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.712794 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e008a429b9e89965376f4032927df6ffc5402b26838c4454a2730a7fd6343b"} err="failed to get container status \"91e008a429b9e89965376f4032927df6ffc5402b26838c4454a2730a7fd6343b\": rpc error: code = NotFound desc = could not find container \"91e008a429b9e89965376f4032927df6ffc5402b26838c4454a2730a7fd6343b\": container with ID starting with 91e008a429b9e89965376f4032927df6ffc5402b26838c4454a2730a7fd6343b not found: ID does not exist" Sep 08 19:06:44.712834 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.712809 2553 scope.go:117] "RemoveContainer" containerID="b4c8f83bad49085565f0ffd7f6f1fdbf515316ec7f05a70eb2eb74e531867fa9" Sep 08 19:06:44.721250 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.721232 2553 scope.go:117] "RemoveContainer" containerID="b4c8f83bad49085565f0ffd7f6f1fdbf515316ec7f05a70eb2eb74e531867fa9" Sep 08 19:06:44.721550 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:06:44.721525 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c8f83bad49085565f0ffd7f6f1fdbf515316ec7f05a70eb2eb74e531867fa9\": container with ID starting with b4c8f83bad49085565f0ffd7f6f1fdbf515316ec7f05a70eb2eb74e531867fa9 not found: ID does not exist" containerID="b4c8f83bad49085565f0ffd7f6f1fdbf515316ec7f05a70eb2eb74e531867fa9" Sep 08 19:06:44.721647 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.721558 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c8f83bad49085565f0ffd7f6f1fdbf515316ec7f05a70eb2eb74e531867fa9"} err="failed to get container status \"b4c8f83bad49085565f0ffd7f6f1fdbf515316ec7f05a70eb2eb74e531867fa9\": rpc error: code = NotFound desc = could not find container \"b4c8f83bad49085565f0ffd7f6f1fdbf515316ec7f05a70eb2eb74e531867fa9\": container with ID starting with b4c8f83bad49085565f0ffd7f6f1fdbf515316ec7f05a70eb2eb74e531867fa9 not found: ID does not exist" Sep 08 19:06:44.744658 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.744634 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa4c6e3a-63a4-45b2-b085-24310123bd1e" path="/var/lib/kubelet/pods/fa4c6e3a-63a4-45b2-b085-24310123bd1e/volumes" Sep 08 19:06:44.788296 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.788272 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bd4061-c9b0-4fd4-a1cf-ec835b614d44-plugins-config-volume\") pod \"35bd4061-c9b0-4fd4-a1cf-ec835b614d44\" (UID: \"35bd4061-c9b0-4fd4-a1cf-ec835b614d44\") " Sep 08 19:06:44.788419 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.788308 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59h97\" (UniqueName: \"kubernetes.io/projected/25813327-9a1c-4b26-8417-58aeba664091-kube-api-access-59h97\") pod \"25813327-9a1c-4b26-8417-58aeba664091\" (UID: \"25813327-9a1c-4b26-8417-58aeba664091\") " Sep 08 19:06:44.788419 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.788395 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5h6l\" (UniqueName: \"kubernetes.io/projected/35bd4061-c9b0-4fd4-a1cf-ec835b614d44-kube-api-access-n5h6l\") pod \"35bd4061-c9b0-4fd4-a1cf-ec835b614d44\" (UID: \"35bd4061-c9b0-4fd4-a1cf-ec835b614d44\") " Sep 08 19:06:44.788506 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.788433 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25813327-9a1c-4b26-8417-58aeba664091-plugins-config-volume\") pod \"25813327-9a1c-4b26-8417-58aeba664091\" (UID: \"25813327-9a1c-4b26-8417-58aeba664091\") " Sep 08 19:06:44.788747 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.788719 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35bd4061-c9b0-4fd4-a1cf-ec835b614d44-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "35bd4061-c9b0-4fd4-a1cf-ec835b614d44" (UID: "35bd4061-c9b0-4fd4-a1cf-ec835b614d44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:06:44.788835 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.788815 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25813327-9a1c-4b26-8417-58aeba664091-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25813327-9a1c-4b26-8417-58aeba664091" (UID: "25813327-9a1c-4b26-8417-58aeba664091"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:06:44.790374 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.790345 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25813327-9a1c-4b26-8417-58aeba664091-kube-api-access-59h97" (OuterVolumeSpecName: "kube-api-access-59h97") pod "25813327-9a1c-4b26-8417-58aeba664091" (UID: "25813327-9a1c-4b26-8417-58aeba664091"). InnerVolumeSpecName "kube-api-access-59h97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:06:44.790443 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.790375 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bd4061-c9b0-4fd4-a1cf-ec835b614d44-kube-api-access-n5h6l" (OuterVolumeSpecName: "kube-api-access-n5h6l") pod "35bd4061-c9b0-4fd4-a1cf-ec835b614d44" (UID: "35bd4061-c9b0-4fd4-a1cf-ec835b614d44"). InnerVolumeSpecName "kube-api-access-n5h6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:06:44.889207 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.889148 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25813327-9a1c-4b26-8417-58aeba664091-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:06:44.889207 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.889171 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bd4061-c9b0-4fd4-a1cf-ec835b614d44-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:06:44.889207 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.889180 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59h97\" (UniqueName: \"kubernetes.io/projected/25813327-9a1c-4b26-8417-58aeba664091-kube-api-access-59h97\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:06:44.889207 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:44.889188 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5h6l\" (UniqueName: \"kubernetes.io/projected/35bd4061-c9b0-4fd4-a1cf-ec835b614d44-kube-api-access-n5h6l\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:06:45.027596 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:45.027564 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr"] Sep 08 19:06:45.034261 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:45.034237 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e77525-699fd6dcc4-dschr"] Sep 08 19:06:45.046851 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:45.046829 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6"] Sep 08 19:06:45.051037 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:45.051011 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e77525-68d9f6c959-wsjm6"] Sep 08 19:06:46.745313 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:46.745278 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25813327-9a1c-4b26-8417-58aeba664091" path="/var/lib/kubelet/pods/25813327-9a1c-4b26-8417-58aeba664091/volumes" Sep 08 19:06:46.745703 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:06:46.745661 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35bd4061-c9b0-4fd4-a1cf-ec835b614d44" path="/var/lib/kubelet/pods/35bd4061-c9b0-4fd4-a1cf-ec835b614d44/volumes" Sep 08 19:07:33.263422 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.263385 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x"] Sep 08 19:07:33.263988 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.263965 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4c6e3a-63a4-45b2-b085-24310123bd1e" containerName="revoke-keys" Sep 08 19:07:33.264071 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.263992 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4c6e3a-63a4-45b2-b085-24310123bd1e" containerName="revoke-keys" Sep 08 19:07:33.264071 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.264016 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25813327-9a1c-4b26-8417-58aeba664091" containerName="payload-pre-processing" Sep 08 19:07:33.264071 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.264025 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="25813327-9a1c-4b26-8417-58aeba664091" containerName="payload-pre-processing" Sep 08 19:07:33.264071 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.264047 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35bd4061-c9b0-4fd4-a1cf-ec835b614d44" containerName="payload-processing" Sep 08 19:07:33.264071 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.264056 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="35bd4061-c9b0-4fd4-a1cf-ec835b614d44" containerName="payload-processing" Sep 08 19:07:33.264329 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.264145 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="25813327-9a1c-4b26-8417-58aeba664091" containerName="payload-pre-processing" Sep 08 19:07:33.264329 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.264162 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4c6e3a-63a4-45b2-b085-24310123bd1e" containerName="revoke-keys" Sep 08 19:07:33.264329 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.264172 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="35bd4061-c9b0-4fd4-a1cf-ec835b614d44" containerName="payload-processing" Sep 08 19:07:33.267295 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.267275 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x" Sep 08 19:07:33.270073 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.270052 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 19:07:33.270260 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.270239 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 19:07:33.276079 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.276057 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x"] Sep 08 19:07:33.427212 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.427178 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbed146-a4aa-47d7-8bf1-36f7b737bc0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x\" (UID: \"7dbed146-a4aa-47d7-8bf1-36f7b737bc0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x" Sep 08 19:07:33.528477 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.528395 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbed146-a4aa-47d7-8bf1-36f7b737bc0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x\" (UID: \"7dbed146-a4aa-47d7-8bf1-36f7b737bc0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x" Sep 08 19:07:33.529002 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.528981 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbed146-a4aa-47d7-8bf1-36f7b737bc0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x\" (UID: \"7dbed146-a4aa-47d7-8bf1-36f7b737bc0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x" Sep 08 19:07:33.579513 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.579480 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x" Sep 08 19:07:33.856737 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.856708 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x"] Sep 08 19:07:33.857871 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:07:33.857846 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbed146_a4aa_47d7_8bf1_36f7b737bc0f.slice/crio-5abbc3ae0683f7ca25b2737d44ac1b8c9950941049356859aca8c29418f4c35d WatchSource:0}: Error finding container 5abbc3ae0683f7ca25b2737d44ac1b8c9950941049356859aca8c29418f4c35d: Status 404 returned error can't find the container with id 5abbc3ae0683f7ca25b2737d44ac1b8c9950941049356859aca8c29418f4c35d Sep 08 19:07:33.988341 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:33.988306 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x" event={"ID":"7dbed146-a4aa-47d7-8bf1-36f7b737bc0f","Type":"ContainerStarted","Data":"5abbc3ae0683f7ca25b2737d44ac1b8c9950941049356859aca8c29418f4c35d"} Sep 08 19:07:34.995935 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:34.995833 2553 generic.go:356] "Generic (PLEG): container finished" podID="7dbed146-a4aa-47d7-8bf1-36f7b737bc0f" containerID="9f5215eb89c879e84f144560d8a799f54fd87d366cfa322eb8d795c7fbdd0697" exitCode=0 Sep 08 19:07:34.996283 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:34.995931 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x" event={"ID":"7dbed146-a4aa-47d7-8bf1-36f7b737bc0f","Type":"ContainerDied","Data":"9f5215eb89c879e84f144560d8a799f54fd87d366cfa322eb8d795c7fbdd0697"} Sep 08 19:07:36.131558 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:36.131535 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x" Sep 08 19:07:36.253467 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:36.253388 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbed146-a4aa-47d7-8bf1-36f7b737bc0f-maas-api-service-ca\") pod \"7dbed146-a4aa-47d7-8bf1-36f7b737bc0f\" (UID: \"7dbed146-a4aa-47d7-8bf1-36f7b737bc0f\") " Sep 08 19:07:36.253785 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:36.253764 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbed146-a4aa-47d7-8bf1-36f7b737bc0f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7dbed146-a4aa-47d7-8bf1-36f7b737bc0f" (UID: "7dbed146-a4aa-47d7-8bf1-36f7b737bc0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:07:36.354546 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:36.354502 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dbed146-a4aa-47d7-8bf1-36f7b737bc0f-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:07:37.008740 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:37.008663 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x" Sep 08 19:07:37.008740 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:37.008684 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x" event={"ID":"7dbed146-a4aa-47d7-8bf1-36f7b737bc0f","Type":"ContainerDied","Data":"5abbc3ae0683f7ca25b2737d44ac1b8c9950941049356859aca8c29418f4c35d"} Sep 08 19:07:37.008740 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:37.008729 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5abbc3ae0683f7ca25b2737d44ac1b8c9950941049356859aca8c29418f4c35d" Sep 08 19:07:43.733710 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:43.733319 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h"] Sep 08 19:07:43.734283 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:43.733771 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" podUID="5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd" containerName="payload-pre-processing" containerID="cri-o://f7840251baf2ad73a4f45c3d3c08632e2daf2a38fe79f079bd578353c33289bf" gracePeriod=30 Sep 08 19:07:43.881439 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:43.881401 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x"] Sep 08 19:07:43.883937 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:43.883906 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1a27c5-qhq4x"] Sep 08 19:07:43.993652 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:43.993580 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" Sep 08 19:07:44.006977 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.006953 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd-plugins-config-volume\") pod \"5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd\" (UID: \"5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd\") " Sep 08 19:07:44.007100 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.007010 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwk92\" (UniqueName: \"kubernetes.io/projected/5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd-kube-api-access-pwk92\") pod \"5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd\" (UID: \"5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd\") " Sep 08 19:07:44.007358 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.007326 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd" (UID: "5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:07:44.008962 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.008942 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd-kube-api-access-pwk92" (OuterVolumeSpecName: "kube-api-access-pwk92") pod "5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd" (UID: "5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd"). InnerVolumeSpecName "kube-api-access-pwk92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:07:44.048689 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.048663 2553 generic.go:356] "Generic (PLEG): container finished" podID="5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd" containerID="f7840251baf2ad73a4f45c3d3c08632e2daf2a38fe79f079bd578353c33289bf" exitCode=0 Sep 08 19:07:44.048800 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.048707 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" Sep 08 19:07:44.048800 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.048738 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" event={"ID":"5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd","Type":"ContainerDied","Data":"f7840251baf2ad73a4f45c3d3c08632e2daf2a38fe79f079bd578353c33289bf"} Sep 08 19:07:44.048800 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.048779 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h" event={"ID":"5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd","Type":"ContainerDied","Data":"57a8ddd710a18e4c9ee4dd39ceaed47d35abe971f8598a0b6453608161aa88ad"} Sep 08 19:07:44.048932 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.048810 2553 scope.go:117] "RemoveContainer" containerID="f7840251baf2ad73a4f45c3d3c08632e2daf2a38fe79f079bd578353c33289bf" Sep 08 19:07:44.058574 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.058557 2553 scope.go:117] "RemoveContainer" containerID="f7840251baf2ad73a4f45c3d3c08632e2daf2a38fe79f079bd578353c33289bf" Sep 08 19:07:44.058848 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:07:44.058826 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7840251baf2ad73a4f45c3d3c08632e2daf2a38fe79f079bd578353c33289bf\": container with ID starting with f7840251baf2ad73a4f45c3d3c08632e2daf2a38fe79f079bd578353c33289bf not found: ID does not exist" containerID="f7840251baf2ad73a4f45c3d3c08632e2daf2a38fe79f079bd578353c33289bf" Sep 08 19:07:44.058912 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.058859 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7840251baf2ad73a4f45c3d3c08632e2daf2a38fe79f079bd578353c33289bf"} err="failed to get container status \"f7840251baf2ad73a4f45c3d3c08632e2daf2a38fe79f079bd578353c33289bf\": rpc error: code = NotFound desc = could not find container \"f7840251baf2ad73a4f45c3d3c08632e2daf2a38fe79f079bd578353c33289bf\": container with ID starting with f7840251baf2ad73a4f45c3d3c08632e2daf2a38fe79f079bd578353c33289bf not found: ID does not exist" Sep 08 19:07:44.071388 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.071354 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h"] Sep 08 19:07:44.075060 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.075040 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a27c5-697d4959cd-dt68h"] Sep 08 19:07:44.107841 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.107818 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:07:44.107841 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.107840 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwk92\" (UniqueName: \"kubernetes.io/projected/5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd-kube-api-access-pwk92\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:07:44.745174 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.745143 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd" path="/var/lib/kubelet/pods/5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd/volumes" Sep 08 19:07:44.745580 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:44.745461 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dbed146-a4aa-47d7-8bf1-36f7b737bc0f" path="/var/lib/kubelet/pods/7dbed146-a4aa-47d7-8bf1-36f7b737bc0f/volumes" Sep 08 19:07:50.625037 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:50.624989 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf"] Sep 08 19:07:50.625541 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:50.625356 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dbed146-a4aa-47d7-8bf1-36f7b737bc0f" containerName="revoke-keys" Sep 08 19:07:50.625541 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:50.625383 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dbed146-a4aa-47d7-8bf1-36f7b737bc0f" containerName="revoke-keys" Sep 08 19:07:50.625541 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:50.625400 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd" containerName="payload-pre-processing" Sep 08 19:07:50.625541 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:50.625406 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd" containerName="payload-pre-processing" Sep 08 19:07:50.625541 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:50.625474 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ebb7468-d6d9-4c86-b8ca-4e6f7c2945fd" containerName="payload-pre-processing" Sep 08 19:07:50.625541 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:50.625482 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dbed146-a4aa-47d7-8bf1-36f7b737bc0f" containerName="revoke-keys" Sep 08 19:07:50.629820 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:50.629791 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf" Sep 08 19:07:50.633182 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:50.633156 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 19:07:50.633307 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:50.633156 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 19:07:50.635849 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:50.635823 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf"] Sep 08 19:07:50.663301 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:50.663273 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b581da4e-f2b0-43a0-a94c-a056bbc5738a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf\" (UID: \"b581da4e-f2b0-43a0-a94c-a056bbc5738a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf" Sep 08 19:07:50.764436 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:50.764406 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b581da4e-f2b0-43a0-a94c-a056bbc5738a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf\" (UID: \"b581da4e-f2b0-43a0-a94c-a056bbc5738a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf" Sep 08 19:07:50.765087 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:50.765068 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b581da4e-f2b0-43a0-a94c-a056bbc5738a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf\" (UID: \"b581da4e-f2b0-43a0-a94c-a056bbc5738a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf" Sep 08 19:07:50.942221 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:50.942137 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf" Sep 08 19:07:51.215860 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:51.215831 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf"] Sep 08 19:07:51.218568 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:07:51.218538 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb581da4e_f2b0_43a0_a94c_a056bbc5738a.slice/crio-d4315decb6dbde6184b1c312b43c65bc6d9cd977530eabeaf7987deff18e4282 WatchSource:0}: Error finding container d4315decb6dbde6184b1c312b43c65bc6d9cd977530eabeaf7987deff18e4282: Status 404 returned error can't find the container with id d4315decb6dbde6184b1c312b43c65bc6d9cd977530eabeaf7987deff18e4282 Sep 08 19:07:52.097422 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:52.097392 2553 generic.go:356] "Generic (PLEG): container finished" podID="b581da4e-f2b0-43a0-a94c-a056bbc5738a" containerID="db4f3a9489711bc1260147b08336026740922c4769c45f5e6a2045a1753a265f" exitCode=0 Sep 08 19:07:52.097796 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:52.097479 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf" event={"ID":"b581da4e-f2b0-43a0-a94c-a056bbc5738a","Type":"ContainerDied","Data":"db4f3a9489711bc1260147b08336026740922c4769c45f5e6a2045a1753a265f"} Sep 08 19:07:52.097796 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:52.097532 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf" event={"ID":"b581da4e-f2b0-43a0-a94c-a056bbc5738a","Type":"ContainerStarted","Data":"d4315decb6dbde6184b1c312b43c65bc6d9cd977530eabeaf7987deff18e4282"} Sep 08 19:07:53.231609 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:53.231586 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf" Sep 08 19:07:53.288134 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:53.288102 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b581da4e-f2b0-43a0-a94c-a056bbc5738a-maas-api-service-ca\") pod \"b581da4e-f2b0-43a0-a94c-a056bbc5738a\" (UID: \"b581da4e-f2b0-43a0-a94c-a056bbc5738a\") " Sep 08 19:07:53.288537 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:53.288508 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b581da4e-f2b0-43a0-a94c-a056bbc5738a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b581da4e-f2b0-43a0-a94c-a056bbc5738a" (UID: "b581da4e-f2b0-43a0-a94c-a056bbc5738a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:07:53.389046 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:53.389014 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b581da4e-f2b0-43a0-a94c-a056bbc5738a-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:07:54.110894 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:54.110860 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf" event={"ID":"b581da4e-f2b0-43a0-a94c-a056bbc5738a","Type":"ContainerDied","Data":"d4315decb6dbde6184b1c312b43c65bc6d9cd977530eabeaf7987deff18e4282"} Sep 08 19:07:54.110894 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:54.110895 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf" Sep 08 19:07:54.111100 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:07:54.110930 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4315decb6dbde6184b1c312b43c65bc6d9cd977530eabeaf7987deff18e4282" Sep 08 19:08:01.378805 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:01.378770 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4001b-29814900-d6z5c"] Sep 08 19:08:01.381436 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:01.381406 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4001b-29814900-d6z5c"] Sep 08 19:08:01.408029 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:01.407986 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm"] Sep 08 19:08:01.408335 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:01.408304 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" podUID="00659bb0-6e3b-4f6a-8816-58abc54f85c9" containerName="payload-processing" containerID="cri-o://6b63a60755d4d703457c1a018229e691d037eabeba579b67e11c9a8eb240a499" gracePeriod=30 Sep 08 19:08:01.571069 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:01.571028 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf"] Sep 08 19:08:01.573256 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:01.573228 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a4001b-jb9rf"] Sep 08 19:08:01.671415 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:01.671387 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" Sep 08 19:08:01.758971 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:01.758937 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00659bb0-6e3b-4f6a-8816-58abc54f85c9-plugins-config-volume\") pod \"00659bb0-6e3b-4f6a-8816-58abc54f85c9\" (UID: \"00659bb0-6e3b-4f6a-8816-58abc54f85c9\") " Sep 08 19:08:01.759149 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:01.759056 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qflv8\" (UniqueName: \"kubernetes.io/projected/00659bb0-6e3b-4f6a-8816-58abc54f85c9-kube-api-access-qflv8\") pod \"00659bb0-6e3b-4f6a-8816-58abc54f85c9\" (UID: \"00659bb0-6e3b-4f6a-8816-58abc54f85c9\") " Sep 08 19:08:01.759285 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:01.759262 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00659bb0-6e3b-4f6a-8816-58abc54f85c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00659bb0-6e3b-4f6a-8816-58abc54f85c9" (UID: "00659bb0-6e3b-4f6a-8816-58abc54f85c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:08:01.761215 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:01.761193 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00659bb0-6e3b-4f6a-8816-58abc54f85c9-kube-api-access-qflv8" (OuterVolumeSpecName: "kube-api-access-qflv8") pod "00659bb0-6e3b-4f6a-8816-58abc54f85c9" (UID: "00659bb0-6e3b-4f6a-8816-58abc54f85c9"). InnerVolumeSpecName "kube-api-access-qflv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:08:01.860676 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:01.860645 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00659bb0-6e3b-4f6a-8816-58abc54f85c9-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:08:01.860676 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:01.860672 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qflv8\" (UniqueName: \"kubernetes.io/projected/00659bb0-6e3b-4f6a-8816-58abc54f85c9-kube-api-access-qflv8\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:08:02.159908 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:02.159872 2553 generic.go:356] "Generic (PLEG): container finished" podID="00659bb0-6e3b-4f6a-8816-58abc54f85c9" containerID="6b63a60755d4d703457c1a018229e691d037eabeba579b67e11c9a8eb240a499" exitCode=0 Sep 08 19:08:02.160085 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:02.159920 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" Sep 08 19:08:02.160085 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:02.159964 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" event={"ID":"00659bb0-6e3b-4f6a-8816-58abc54f85c9","Type":"ContainerDied","Data":"6b63a60755d4d703457c1a018229e691d037eabeba579b67e11c9a8eb240a499"} Sep 08 19:08:02.160085 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:02.160024 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm" event={"ID":"00659bb0-6e3b-4f6a-8816-58abc54f85c9","Type":"ContainerDied","Data":"bd4274e156e824b7ddb2abbc2d33406afe994203a429a14fe3365f94007f73a5"} Sep 08 19:08:02.160085 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:02.160046 2553 scope.go:117] "RemoveContainer" containerID="6b63a60755d4d703457c1a018229e691d037eabeba579b67e11c9a8eb240a499" Sep 08 19:08:02.170217 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:02.170197 2553 scope.go:117] "RemoveContainer" containerID="6b63a60755d4d703457c1a018229e691d037eabeba579b67e11c9a8eb240a499" Sep 08 19:08:02.170505 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:08:02.170487 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b63a60755d4d703457c1a018229e691d037eabeba579b67e11c9a8eb240a499\": container with ID starting with 6b63a60755d4d703457c1a018229e691d037eabeba579b67e11c9a8eb240a499 not found: ID does not exist" containerID="6b63a60755d4d703457c1a018229e691d037eabeba579b67e11c9a8eb240a499" Sep 08 19:08:02.170552 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:02.170513 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b63a60755d4d703457c1a018229e691d037eabeba579b67e11c9a8eb240a499"} err="failed to get container status \"6b63a60755d4d703457c1a018229e691d037eabeba579b67e11c9a8eb240a499\": rpc error: code = NotFound desc = could not find container \"6b63a60755d4d703457c1a018229e691d037eabeba579b67e11c9a8eb240a499\": container with ID starting with 6b63a60755d4d703457c1a018229e691d037eabeba579b67e11c9a8eb240a499 not found: ID does not exist" Sep 08 19:08:02.191505 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:02.191478 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm"] Sep 08 19:08:02.193916 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:02.193892 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4001b-fdd9484bc-nkwwm"] Sep 08 19:08:02.744394 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:02.744348 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00659bb0-6e3b-4f6a-8816-58abc54f85c9" path="/var/lib/kubelet/pods/00659bb0-6e3b-4f6a-8816-58abc54f85c9/volumes" Sep 08 19:08:02.744771 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:02.744672 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22456267-acb6-466a-a1b8-e4123fb861ac" path="/var/lib/kubelet/pods/22456267-acb6-466a-a1b8-e4123fb861ac/volumes" Sep 08 19:08:02.744962 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:02.744949 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b581da4e-f2b0-43a0-a94c-a056bbc5738a" path="/var/lib/kubelet/pods/b581da4e-f2b0-43a0-a94c-a056bbc5738a/volumes" Sep 08 19:08:08.977501 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:08.977464 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2"] Sep 08 19:08:08.979791 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:08.977838 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00659bb0-6e3b-4f6a-8816-58abc54f85c9" containerName="payload-processing" Sep 08 19:08:08.979791 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:08.977849 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="00659bb0-6e3b-4f6a-8816-58abc54f85c9" containerName="payload-processing" Sep 08 19:08:08.979791 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:08.977859 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b581da4e-f2b0-43a0-a94c-a056bbc5738a" containerName="revoke-keys" Sep 08 19:08:08.979791 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:08.977865 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b581da4e-f2b0-43a0-a94c-a056bbc5738a" containerName="revoke-keys" Sep 08 19:08:08.979791 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:08.977934 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="00659bb0-6e3b-4f6a-8816-58abc54f85c9" containerName="payload-processing" Sep 08 19:08:08.979791 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:08.977942 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b581da4e-f2b0-43a0-a94c-a056bbc5738a" containerName="revoke-keys" Sep 08 19:08:08.980780 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:08.980763 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2" Sep 08 19:08:08.983226 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:08.983205 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 19:08:08.983349 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:08.983328 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 19:08:08.987826 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:08.987800 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2"] Sep 08 19:08:09.125746 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:09.125711 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d581987b-8858-47e4-ad13-03e4922fb9e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2\" (UID: \"d581987b-8858-47e4-ad13-03e4922fb9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2" Sep 08 19:08:09.226668 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:09.226630 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d581987b-8858-47e4-ad13-03e4922fb9e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2\" (UID: \"d581987b-8858-47e4-ad13-03e4922fb9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2" Sep 08 19:08:09.227243 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:09.227222 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d581987b-8858-47e4-ad13-03e4922fb9e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2\" (UID: \"d581987b-8858-47e4-ad13-03e4922fb9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2" Sep 08 19:08:09.292870 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:09.292794 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2" Sep 08 19:08:09.562988 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:09.562959 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2"] Sep 08 19:08:09.564570 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:08:09.564542 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd581987b_8858_47e4_ad13_03e4922fb9e0.slice/crio-a70e8f0ff61be444abbf0806537d142d14489ca9c30f9566478ba6a9b75741ca WatchSource:0}: Error finding container a70e8f0ff61be444abbf0806537d142d14489ca9c30f9566478ba6a9b75741ca: Status 404 returned error can't find the container with id a70e8f0ff61be444abbf0806537d142d14489ca9c30f9566478ba6a9b75741ca Sep 08 19:08:10.208809 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:10.208771 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2" event={"ID":"d581987b-8858-47e4-ad13-03e4922fb9e0","Type":"ContainerStarted","Data":"e7f9c9a0e49d63c15f2b2853d97ba5a4db1f5c1e028c60dafc5579051b6843ea"} Sep 08 19:08:10.209254 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:10.208821 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2" event={"ID":"d581987b-8858-47e4-ad13-03e4922fb9e0","Type":"ContainerStarted","Data":"a70e8f0ff61be444abbf0806537d142d14489ca9c30f9566478ba6a9b75741ca"} Sep 08 19:08:10.225051 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:10.225005 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2" podStartSLOduration=2.224992275 podStartE2EDuration="2.224992275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:08:10.222896875 +0000 UTC m=+2025.928785107" watchObservedRunningTime="2026-09-08 19:08:10.224992275 +0000 UTC m=+2025.930880531" Sep 08 19:08:11.216121 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:11.216082 2553 generic.go:356] "Generic (PLEG): container finished" podID="d581987b-8858-47e4-ad13-03e4922fb9e0" containerID="e7f9c9a0e49d63c15f2b2853d97ba5a4db1f5c1e028c60dafc5579051b6843ea" exitCode=0 Sep 08 19:08:11.216563 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:11.216168 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2" event={"ID":"d581987b-8858-47e4-ad13-03e4922fb9e0","Type":"ContainerDied","Data":"e7f9c9a0e49d63c15f2b2853d97ba5a4db1f5c1e028c60dafc5579051b6843ea"} Sep 08 19:08:12.352441 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:12.352416 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2" Sep 08 19:08:12.454511 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:12.454478 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d581987b-8858-47e4-ad13-03e4922fb9e0-maas-api-service-ca\") pod \"d581987b-8858-47e4-ad13-03e4922fb9e0\" (UID: \"d581987b-8858-47e4-ad13-03e4922fb9e0\") " Sep 08 19:08:12.454823 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:12.454799 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d581987b-8858-47e4-ad13-03e4922fb9e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d581987b-8858-47e4-ad13-03e4922fb9e0" (UID: "d581987b-8858-47e4-ad13-03e4922fb9e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:08:12.555519 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:12.555449 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d581987b-8858-47e4-ad13-03e4922fb9e0-maas-api-service-ca\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:08:13.228899 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:13.228869 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2" Sep 08 19:08:13.228899 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:13.228882 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2" event={"ID":"d581987b-8858-47e4-ad13-03e4922fb9e0","Type":"ContainerDied","Data":"a70e8f0ff61be444abbf0806537d142d14489ca9c30f9566478ba6a9b75741ca"} Sep 08 19:08:13.229118 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:13.228936 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a70e8f0ff61be444abbf0806537d142d14489ca9c30f9566478ba6a9b75741ca" Sep 08 19:08:19.549142 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:19.549107 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-525196-29814900-fgrmv"] Sep 08 19:08:19.551419 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:19.551394 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-525196-29814900-fgrmv"] Sep 08 19:08:19.573122 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:19.573089 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7"] Sep 08 19:08:19.573486 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:19.573455 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" podUID="a282b530-9233-4a7c-9104-020177cef1aa" containerName="payload-processing" containerID="cri-o://eeda73dd3d94df67dbca5f3628bfb8ec7b6f18475d4226986c35a59203b7966f" gracePeriod=30 Sep 08 19:08:19.579149 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:19.579121 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk"] Sep 08 19:08:19.579974 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:19.579328 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" podUID="dde2aafc-0c41-47d1-b928-9541ecb0dd17" containerName="payload-pre-processing" containerID="cri-o://06ce1b098daf87a1e7df736f6b63d25a219d5a811f2549a9cd08a4781247da1f" gracePeriod=30 Sep 08 19:08:19.728933 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:19.728781 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2"] Sep 08 19:08:19.731956 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:19.731930 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-525196-2l6b2"] Sep 08 19:08:19.853233 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:19.853209 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" Sep 08 19:08:19.855943 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:19.855919 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" Sep 08 19:08:20.017429 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.017397 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a282b530-9233-4a7c-9104-020177cef1aa-plugins-config-volume\") pod \"a282b530-9233-4a7c-9104-020177cef1aa\" (UID: \"a282b530-9233-4a7c-9104-020177cef1aa\") " Sep 08 19:08:20.017593 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.017456 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde2aafc-0c41-47d1-b928-9541ecb0dd17-plugins-config-volume\") pod \"dde2aafc-0c41-47d1-b928-9541ecb0dd17\" (UID: \"dde2aafc-0c41-47d1-b928-9541ecb0dd17\") " Sep 08 19:08:20.017593 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.017503 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxzgv\" (UniqueName: \"kubernetes.io/projected/a282b530-9233-4a7c-9104-020177cef1aa-kube-api-access-bxzgv\") pod \"a282b530-9233-4a7c-9104-020177cef1aa\" (UID: \"a282b530-9233-4a7c-9104-020177cef1aa\") " Sep 08 19:08:20.017593 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.017525 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfqpv\" (UniqueName: \"kubernetes.io/projected/dde2aafc-0c41-47d1-b928-9541ecb0dd17-kube-api-access-zfqpv\") pod \"dde2aafc-0c41-47d1-b928-9541ecb0dd17\" (UID: \"dde2aafc-0c41-47d1-b928-9541ecb0dd17\") " Sep 08 19:08:20.017801 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.017771 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde2aafc-0c41-47d1-b928-9541ecb0dd17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dde2aafc-0c41-47d1-b928-9541ecb0dd17" (UID: "dde2aafc-0c41-47d1-b928-9541ecb0dd17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:08:20.017915 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.017809 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a282b530-9233-4a7c-9104-020177cef1aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a282b530-9233-4a7c-9104-020177cef1aa" (UID: "a282b530-9233-4a7c-9104-020177cef1aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 19:08:20.019613 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.019583 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde2aafc-0c41-47d1-b928-9541ecb0dd17-kube-api-access-zfqpv" (OuterVolumeSpecName: "kube-api-access-zfqpv") pod "dde2aafc-0c41-47d1-b928-9541ecb0dd17" (UID: "dde2aafc-0c41-47d1-b928-9541ecb0dd17"). InnerVolumeSpecName "kube-api-access-zfqpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:08:20.019613 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.019593 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a282b530-9233-4a7c-9104-020177cef1aa-kube-api-access-bxzgv" (OuterVolumeSpecName: "kube-api-access-bxzgv") pod "a282b530-9233-4a7c-9104-020177cef1aa" (UID: "a282b530-9233-4a7c-9104-020177cef1aa"). InnerVolumeSpecName "kube-api-access-bxzgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:08:20.118693 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.118611 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a282b530-9233-4a7c-9104-020177cef1aa-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:08:20.118693 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.118640 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde2aafc-0c41-47d1-b928-9541ecb0dd17-plugins-config-volume\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:08:20.118693 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.118652 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxzgv\" (UniqueName: \"kubernetes.io/projected/a282b530-9233-4a7c-9104-020177cef1aa-kube-api-access-bxzgv\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:08:20.118693 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.118663 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfqpv\" (UniqueName: \"kubernetes.io/projected/dde2aafc-0c41-47d1-b928-9541ecb0dd17-kube-api-access-zfqpv\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:08:20.270853 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.270821 2553 generic.go:356] "Generic (PLEG): container finished" podID="a282b530-9233-4a7c-9104-020177cef1aa" containerID="eeda73dd3d94df67dbca5f3628bfb8ec7b6f18475d4226986c35a59203b7966f" exitCode=0 Sep 08 19:08:20.271030 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.270862 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" Sep 08 19:08:20.271030 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.270900 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" event={"ID":"a282b530-9233-4a7c-9104-020177cef1aa","Type":"ContainerDied","Data":"eeda73dd3d94df67dbca5f3628bfb8ec7b6f18475d4226986c35a59203b7966f"} Sep 08 19:08:20.271030 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.270958 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7" event={"ID":"a282b530-9233-4a7c-9104-020177cef1aa","Type":"ContainerDied","Data":"ccc742d34f9f1608d0d07cc2f310864ee297d1462532b85f90b0a6b18a359d4e"} Sep 08 19:08:20.271030 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.270992 2553 scope.go:117] "RemoveContainer" containerID="eeda73dd3d94df67dbca5f3628bfb8ec7b6f18475d4226986c35a59203b7966f" Sep 08 19:08:20.272490 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.272469 2553 generic.go:356] "Generic (PLEG): container finished" podID="dde2aafc-0c41-47d1-b928-9541ecb0dd17" containerID="06ce1b098daf87a1e7df736f6b63d25a219d5a811f2549a9cd08a4781247da1f" exitCode=0 Sep 08 19:08:20.272600 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.272514 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" Sep 08 19:08:20.272667 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.272606 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" event={"ID":"dde2aafc-0c41-47d1-b928-9541ecb0dd17","Type":"ContainerDied","Data":"06ce1b098daf87a1e7df736f6b63d25a219d5a811f2549a9cd08a4781247da1f"} Sep 08 19:08:20.272725 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.272650 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk" event={"ID":"dde2aafc-0c41-47d1-b928-9541ecb0dd17","Type":"ContainerDied","Data":"6b18673f8b82cd6f5886ed3da71ed997b434ab4ed7efddeea470a35be8722bfa"} Sep 08 19:08:20.284971 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.284951 2553 scope.go:117] "RemoveContainer" containerID="eeda73dd3d94df67dbca5f3628bfb8ec7b6f18475d4226986c35a59203b7966f" Sep 08 19:08:20.285256 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:08:20.285232 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeda73dd3d94df67dbca5f3628bfb8ec7b6f18475d4226986c35a59203b7966f\": container with ID starting with eeda73dd3d94df67dbca5f3628bfb8ec7b6f18475d4226986c35a59203b7966f not found: ID does not exist" containerID="eeda73dd3d94df67dbca5f3628bfb8ec7b6f18475d4226986c35a59203b7966f" Sep 08 19:08:20.285321 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.285267 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeda73dd3d94df67dbca5f3628bfb8ec7b6f18475d4226986c35a59203b7966f"} err="failed to get container status \"eeda73dd3d94df67dbca5f3628bfb8ec7b6f18475d4226986c35a59203b7966f\": rpc error: code = NotFound desc = could not find container \"eeda73dd3d94df67dbca5f3628bfb8ec7b6f18475d4226986c35a59203b7966f\": container with ID starting with eeda73dd3d94df67dbca5f3628bfb8ec7b6f18475d4226986c35a59203b7966f not found: ID does not exist" Sep 08 19:08:20.285321 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.285291 2553 scope.go:117] "RemoveContainer" containerID="06ce1b098daf87a1e7df736f6b63d25a219d5a811f2549a9cd08a4781247da1f" Sep 08 19:08:20.293511 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.293501 2553 scope.go:117] "RemoveContainer" containerID="06ce1b098daf87a1e7df736f6b63d25a219d5a811f2549a9cd08a4781247da1f" Sep 08 19:08:20.293780 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:08:20.293759 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ce1b098daf87a1e7df736f6b63d25a219d5a811f2549a9cd08a4781247da1f\": container with ID starting with 06ce1b098daf87a1e7df736f6b63d25a219d5a811f2549a9cd08a4781247da1f not found: ID does not exist" containerID="06ce1b098daf87a1e7df736f6b63d25a219d5a811f2549a9cd08a4781247da1f" Sep 08 19:08:20.293816 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.293787 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ce1b098daf87a1e7df736f6b63d25a219d5a811f2549a9cd08a4781247da1f"} err="failed to get container status \"06ce1b098daf87a1e7df736f6b63d25a219d5a811f2549a9cd08a4781247da1f\": rpc error: code = NotFound desc = could not find container \"06ce1b098daf87a1e7df736f6b63d25a219d5a811f2549a9cd08a4781247da1f\": container with ID starting with 06ce1b098daf87a1e7df736f6b63d25a219d5a811f2549a9cd08a4781247da1f not found: ID does not exist" Sep 08 19:08:20.306530 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.306507 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7"] Sep 08 19:08:20.320715 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.320694 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-525196-55769cf64f-sh4t7"] Sep 08 19:08:20.337708 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.337684 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk"] Sep 08 19:08:20.358773 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.358749 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-525196-87cdc9946-5r8wk"] Sep 08 19:08:20.745155 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.745122 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a282b530-9233-4a7c-9104-020177cef1aa" path="/var/lib/kubelet/pods/a282b530-9233-4a7c-9104-020177cef1aa/volumes" Sep 08 19:08:20.745658 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.745636 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d581987b-8858-47e4-ad13-03e4922fb9e0" path="/var/lib/kubelet/pods/d581987b-8858-47e4-ad13-03e4922fb9e0/volumes" Sep 08 19:08:20.746051 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.746031 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dde2aafc-0c41-47d1-b928-9541ecb0dd17" path="/var/lib/kubelet/pods/dde2aafc-0c41-47d1-b928-9541ecb0dd17/volumes" Sep 08 19:08:20.746338 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:20.746321 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed71a480-4e6b-4247-9dad-4c66e270066f" path="/var/lib/kubelet/pods/ed71a480-4e6b-4247-9dad-4c66e270066f/volumes" Sep 08 19:08:25.199387 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:25.199336 2553 scope.go:117] "RemoveContainer" containerID="d867dba17242da94e291752c08a45b964f19eda72188e56e24ca149e8c783c08" Sep 08 19:08:25.208421 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:25.208401 2553 scope.go:117] "RemoveContainer" containerID="2e7104f97d3bfea34482865a2a473c8442af15974d962d28ec7f1945fb269872" Sep 08 19:08:28.848039 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:28.848004 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c8c7d87-vxscg"] Sep 08 19:08:29.329695 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:29.329638 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68c8c7d87-vxscg" podUID="e3767535-1475-4933-8594-3e2a79ce4b2b" containerName="manager" containerID="cri-o://c213b765b23cf29acb7406653a945ed55d4ca2fe2a9d197af2108e487ebad2e5" gracePeriod=30 Sep 08 19:08:29.575820 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:29.575790 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8c7d87-vxscg" Sep 08 19:08:29.704062 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:29.704029 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3767535-1475-4933-8594-3e2a79ce4b2b-webhook-cert\") pod \"e3767535-1475-4933-8594-3e2a79ce4b2b\" (UID: \"e3767535-1475-4933-8594-3e2a79ce4b2b\") " Sep 08 19:08:29.704242 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:29.704106 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kfk5\" (UniqueName: \"kubernetes.io/projected/e3767535-1475-4933-8594-3e2a79ce4b2b-kube-api-access-8kfk5\") pod \"e3767535-1475-4933-8594-3e2a79ce4b2b\" (UID: \"e3767535-1475-4933-8594-3e2a79ce4b2b\") " Sep 08 19:08:29.704242 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:29.704220 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3767535-1475-4933-8594-3e2a79ce4b2b-metrics-certs\") pod \"e3767535-1475-4933-8594-3e2a79ce4b2b\" (UID: \"e3767535-1475-4933-8594-3e2a79ce4b2b\") " Sep 08 19:08:29.706109 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:29.706082 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3767535-1475-4933-8594-3e2a79ce4b2b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e3767535-1475-4933-8594-3e2a79ce4b2b" (UID: "e3767535-1475-4933-8594-3e2a79ce4b2b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 19:08:29.706288 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:29.706265 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3767535-1475-4933-8594-3e2a79ce4b2b-kube-api-access-8kfk5" (OuterVolumeSpecName: "kube-api-access-8kfk5") pod "e3767535-1475-4933-8594-3e2a79ce4b2b" (UID: "e3767535-1475-4933-8594-3e2a79ce4b2b"). InnerVolumeSpecName "kube-api-access-8kfk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:08:29.706352 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:29.706316 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3767535-1475-4933-8594-3e2a79ce4b2b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e3767535-1475-4933-8594-3e2a79ce4b2b" (UID: "e3767535-1475-4933-8594-3e2a79ce4b2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 19:08:29.805783 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:29.805756 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kfk5\" (UniqueName: \"kubernetes.io/projected/e3767535-1475-4933-8594-3e2a79ce4b2b-kube-api-access-8kfk5\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:08:29.805783 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:29.805779 2553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3767535-1475-4933-8594-3e2a79ce4b2b-metrics-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:08:29.805783 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:29.805788 2553 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e3767535-1475-4933-8594-3e2a79ce4b2b-webhook-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:08:30.335923 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:30.335888 2553 generic.go:356] "Generic (PLEG): container finished" podID="e3767535-1475-4933-8594-3e2a79ce4b2b" containerID="c213b765b23cf29acb7406653a945ed55d4ca2fe2a9d197af2108e487ebad2e5" exitCode=0 Sep 08 19:08:30.336359 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:30.335937 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8c7d87-vxscg" Sep 08 19:08:30.336359 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:30.335977 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8c7d87-vxscg" event={"ID":"e3767535-1475-4933-8594-3e2a79ce4b2b","Type":"ContainerDied","Data":"c213b765b23cf29acb7406653a945ed55d4ca2fe2a9d197af2108e487ebad2e5"} Sep 08 19:08:30.336359 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:30.336033 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8c7d87-vxscg" event={"ID":"e3767535-1475-4933-8594-3e2a79ce4b2b","Type":"ContainerDied","Data":"ee23ec0e0d94aab8130388b7b4383491989313ae0c150df9444981c26f7cc43f"} Sep 08 19:08:30.336359 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:30.336056 2553 scope.go:117] "RemoveContainer" containerID="c213b765b23cf29acb7406653a945ed55d4ca2fe2a9d197af2108e487ebad2e5" Sep 08 19:08:30.346047 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:30.346029 2553 scope.go:117] "RemoveContainer" containerID="c213b765b23cf29acb7406653a945ed55d4ca2fe2a9d197af2108e487ebad2e5" Sep 08 19:08:30.346296 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:08:30.346277 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c213b765b23cf29acb7406653a945ed55d4ca2fe2a9d197af2108e487ebad2e5\": container with ID starting with c213b765b23cf29acb7406653a945ed55d4ca2fe2a9d197af2108e487ebad2e5 not found: ID does not exist" containerID="c213b765b23cf29acb7406653a945ed55d4ca2fe2a9d197af2108e487ebad2e5" Sep 08 19:08:30.346358 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:30.346309 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c213b765b23cf29acb7406653a945ed55d4ca2fe2a9d197af2108e487ebad2e5"} err="failed to get container status \"c213b765b23cf29acb7406653a945ed55d4ca2fe2a9d197af2108e487ebad2e5\": rpc error: code = NotFound desc = could not find container \"c213b765b23cf29acb7406653a945ed55d4ca2fe2a9d197af2108e487ebad2e5\": container with ID starting with c213b765b23cf29acb7406653a945ed55d4ca2fe2a9d197af2108e487ebad2e5 not found: ID does not exist" Sep 08 19:08:30.358289 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:30.358265 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c8c7d87-vxscg"] Sep 08 19:08:30.363735 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:30.363711 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68c8c7d87-vxscg"] Sep 08 19:08:30.745222 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:08:30.745188 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3767535-1475-4933-8594-3e2a79ce4b2b" path="/var/lib/kubelet/pods/e3767535-1475-4933-8594-3e2a79ce4b2b/volumes" Sep 08 19:09:25.012963 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:25.012932 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 19:09:25.016786 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:25.016765 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 19:09:25.020297 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:25.020277 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 19:09:25.023809 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:25.023794 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 19:09:25.250379 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:25.250336 2553 scope.go:117] "RemoveContainer" containerID="e9a4446c56d06362214ae726cbd33e1b874bef1f029bec9bc825fcd2a1e30e48" Sep 08 19:09:31.164706 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.164666 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68c8c7d87-prjgv"] Sep 08 19:09:31.165173 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.165026 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3767535-1475-4933-8594-3e2a79ce4b2b" containerName="manager" Sep 08 19:09:31.165173 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.165041 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3767535-1475-4933-8594-3e2a79ce4b2b" containerName="manager" Sep 08 19:09:31.165173 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.165049 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dde2aafc-0c41-47d1-b928-9541ecb0dd17" containerName="payload-pre-processing" Sep 08 19:09:31.165173 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.165054 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="dde2aafc-0c41-47d1-b928-9541ecb0dd17" containerName="payload-pre-processing" Sep 08 19:09:31.165173 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.165066 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a282b530-9233-4a7c-9104-020177cef1aa" containerName="payload-processing" Sep 08 19:09:31.165173 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.165072 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a282b530-9233-4a7c-9104-020177cef1aa" containerName="payload-processing" Sep 08 19:09:31.165173 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.165088 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d581987b-8858-47e4-ad13-03e4922fb9e0" containerName="revoke-keys" Sep 08 19:09:31.165173 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.165093 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d581987b-8858-47e4-ad13-03e4922fb9e0" containerName="revoke-keys" Sep 08 19:09:31.165173 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.165150 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a282b530-9233-4a7c-9104-020177cef1aa" containerName="payload-processing" Sep 08 19:09:31.165173 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.165160 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d581987b-8858-47e4-ad13-03e4922fb9e0" containerName="revoke-keys" Sep 08 19:09:31.165173 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.165167 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="dde2aafc-0c41-47d1-b928-9541ecb0dd17" containerName="payload-pre-processing" Sep 08 19:09:31.165173 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.165174 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3767535-1475-4933-8594-3e2a79ce4b2b" containerName="manager" Sep 08 19:09:31.167598 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.167581 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8c7d87-prjgv" Sep 08 19:09:31.171688 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.171659 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 19:09:31.171688 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.171677 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 19:09:31.171905 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.171689 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 19:09:31.171905 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.171667 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pxxwb\"" Sep 08 19:09:31.178425 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.178403 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c8c7d87-prjgv"] Sep 08 19:09:31.218665 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.218637 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d09c6617-3e03-46e8-a313-100512596ca0-webhook-cert\") pod \"maas-controller-68c8c7d87-prjgv\" (UID: \"d09c6617-3e03-46e8-a313-100512596ca0\") " pod="opendatahub/maas-controller-68c8c7d87-prjgv" Sep 08 19:09:31.218784 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.218676 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d09c6617-3e03-46e8-a313-100512596ca0-metrics-certs\") pod \"maas-controller-68c8c7d87-prjgv\" (UID: \"d09c6617-3e03-46e8-a313-100512596ca0\") " pod="opendatahub/maas-controller-68c8c7d87-prjgv" Sep 08 19:09:31.218784 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.218709 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rct8\" (UniqueName: \"kubernetes.io/projected/d09c6617-3e03-46e8-a313-100512596ca0-kube-api-access-9rct8\") pod \"maas-controller-68c8c7d87-prjgv\" (UID: \"d09c6617-3e03-46e8-a313-100512596ca0\") " pod="opendatahub/maas-controller-68c8c7d87-prjgv" Sep 08 19:09:31.319154 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.319125 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d09c6617-3e03-46e8-a313-100512596ca0-webhook-cert\") pod \"maas-controller-68c8c7d87-prjgv\" (UID: \"d09c6617-3e03-46e8-a313-100512596ca0\") " pod="opendatahub/maas-controller-68c8c7d87-prjgv" Sep 08 19:09:31.319304 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.319164 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d09c6617-3e03-46e8-a313-100512596ca0-metrics-certs\") pod \"maas-controller-68c8c7d87-prjgv\" (UID: \"d09c6617-3e03-46e8-a313-100512596ca0\") " pod="opendatahub/maas-controller-68c8c7d87-prjgv" Sep 08 19:09:31.319304 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.319194 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rct8\" (UniqueName: \"kubernetes.io/projected/d09c6617-3e03-46e8-a313-100512596ca0-kube-api-access-9rct8\") pod \"maas-controller-68c8c7d87-prjgv\" (UID: \"d09c6617-3e03-46e8-a313-100512596ca0\") " pod="opendatahub/maas-controller-68c8c7d87-prjgv" Sep 08 19:09:31.321646 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.321625 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d09c6617-3e03-46e8-a313-100512596ca0-webhook-cert\") pod \"maas-controller-68c8c7d87-prjgv\" (UID: \"d09c6617-3e03-46e8-a313-100512596ca0\") " pod="opendatahub/maas-controller-68c8c7d87-prjgv" Sep 08 19:09:31.321709 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.321681 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d09c6617-3e03-46e8-a313-100512596ca0-metrics-certs\") pod \"maas-controller-68c8c7d87-prjgv\" (UID: \"d09c6617-3e03-46e8-a313-100512596ca0\") " pod="opendatahub/maas-controller-68c8c7d87-prjgv" Sep 08 19:09:31.327174 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.327148 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rct8\" (UniqueName: \"kubernetes.io/projected/d09c6617-3e03-46e8-a313-100512596ca0-kube-api-access-9rct8\") pod \"maas-controller-68c8c7d87-prjgv\" (UID: \"d09c6617-3e03-46e8-a313-100512596ca0\") " pod="opendatahub/maas-controller-68c8c7d87-prjgv" Sep 08 19:09:31.478993 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.478929 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8c7d87-prjgv" Sep 08 19:09:31.759816 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.759782 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c8c7d87-prjgv"] Sep 08 19:09:31.760969 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:09:31.760944 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09c6617_3e03_46e8_a313_100512596ca0.slice/crio-3f1b002857777ab0e59e01f0b56af63315ae6a241bf6abf089fe7addeb1fa4c4 WatchSource:0}: Error finding container 3f1b002857777ab0e59e01f0b56af63315ae6a241bf6abf089fe7addeb1fa4c4: Status 404 returned error can't find the container with id 3f1b002857777ab0e59e01f0b56af63315ae6a241bf6abf089fe7addeb1fa4c4 Sep 08 19:09:31.762397 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:31.762378 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 19:09:32.696774 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:32.696735 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8c7d87-prjgv" event={"ID":"d09c6617-3e03-46e8-a313-100512596ca0","Type":"ContainerStarted","Data":"3e350895730625185c8ebd16cf5fe34ed41476b4292fa393d1313a5f6f6557a4"} Sep 08 19:09:32.697195 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:32.696790 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8c7d87-prjgv" event={"ID":"d09c6617-3e03-46e8-a313-100512596ca0","Type":"ContainerStarted","Data":"3f1b002857777ab0e59e01f0b56af63315ae6a241bf6abf089fe7addeb1fa4c4"} Sep 08 19:09:32.697195 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:32.696818 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68c8c7d87-prjgv" Sep 08 19:09:32.714156 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:32.714107 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68c8c7d87-prjgv" podStartSLOduration=1.2598497229999999 podStartE2EDuration="1.714093466s" totalImagesPullingTime="454.243743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:09:32.71234141 +0000 UTC m=+2108.418229633" watchObservedRunningTime="2026-09-08 19:09:32.714093466 +0000 UTC m=+2108.419981736" Sep 08 19:09:43.704660 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:43.704630 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68c8c7d87-prjgv" Sep 08 19:09:53.034799 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.034761 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c8c7d87-prjgv"] Sep 08 19:09:53.035210 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.035003 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68c8c7d87-prjgv" podUID="d09c6617-3e03-46e8-a313-100512596ca0" containerName="manager" containerID="cri-o://3e350895730625185c8ebd16cf5fe34ed41476b4292fa393d1313a5f6f6557a4" gracePeriod=30 Sep 08 19:09:53.283964 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.283935 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8c7d87-prjgv" Sep 08 19:09:53.411023 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.410995 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d09c6617-3e03-46e8-a313-100512596ca0-metrics-certs\") pod \"d09c6617-3e03-46e8-a313-100512596ca0\" (UID: \"d09c6617-3e03-46e8-a313-100512596ca0\") " Sep 08 19:09:53.411190 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.411038 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d09c6617-3e03-46e8-a313-100512596ca0-webhook-cert\") pod \"d09c6617-3e03-46e8-a313-100512596ca0\" (UID: \"d09c6617-3e03-46e8-a313-100512596ca0\") " Sep 08 19:09:53.411190 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.411095 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rct8\" (UniqueName: \"kubernetes.io/projected/d09c6617-3e03-46e8-a313-100512596ca0-kube-api-access-9rct8\") pod \"d09c6617-3e03-46e8-a313-100512596ca0\" (UID: \"d09c6617-3e03-46e8-a313-100512596ca0\") " Sep 08 19:09:53.413114 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.413075 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d09c6617-3e03-46e8-a313-100512596ca0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d09c6617-3e03-46e8-a313-100512596ca0" (UID: "d09c6617-3e03-46e8-a313-100512596ca0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 19:09:53.413236 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.413158 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d09c6617-3e03-46e8-a313-100512596ca0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d09c6617-3e03-46e8-a313-100512596ca0" (UID: "d09c6617-3e03-46e8-a313-100512596ca0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 19:09:53.413236 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.413206 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09c6617-3e03-46e8-a313-100512596ca0-kube-api-access-9rct8" (OuterVolumeSpecName: "kube-api-access-9rct8") pod "d09c6617-3e03-46e8-a313-100512596ca0" (UID: "d09c6617-3e03-46e8-a313-100512596ca0"). InnerVolumeSpecName "kube-api-access-9rct8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 19:09:53.511777 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.511744 2553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d09c6617-3e03-46e8-a313-100512596ca0-metrics-certs\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:09:53.511777 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.511771 2553 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d09c6617-3e03-46e8-a313-100512596ca0-webhook-cert\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:09:53.511963 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.511783 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rct8\" (UniqueName: \"kubernetes.io/projected/d09c6617-3e03-46e8-a313-100512596ca0-kube-api-access-9rct8\") on node \"ip-10-0-132-43.ec2.internal\" DevicePath \"\"" Sep 08 19:09:53.809092 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.808998 2553 generic.go:356] "Generic (PLEG): container finished" podID="d09c6617-3e03-46e8-a313-100512596ca0" containerID="3e350895730625185c8ebd16cf5fe34ed41476b4292fa393d1313a5f6f6557a4" exitCode=0 Sep 08 19:09:53.809092 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.809042 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8c7d87-prjgv" Sep 08 19:09:53.809290 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.809040 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8c7d87-prjgv" event={"ID":"d09c6617-3e03-46e8-a313-100512596ca0","Type":"ContainerDied","Data":"3e350895730625185c8ebd16cf5fe34ed41476b4292fa393d1313a5f6f6557a4"} Sep 08 19:09:53.809290 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.809165 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8c7d87-prjgv" event={"ID":"d09c6617-3e03-46e8-a313-100512596ca0","Type":"ContainerDied","Data":"3f1b002857777ab0e59e01f0b56af63315ae6a241bf6abf089fe7addeb1fa4c4"} Sep 08 19:09:53.809290 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.809186 2553 scope.go:117] "RemoveContainer" containerID="3e350895730625185c8ebd16cf5fe34ed41476b4292fa393d1313a5f6f6557a4" Sep 08 19:09:53.821115 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.821092 2553 scope.go:117] "RemoveContainer" containerID="3e350895730625185c8ebd16cf5fe34ed41476b4292fa393d1313a5f6f6557a4" Sep 08 19:09:53.821558 ip-10-0-132-43 kubenswrapper[2553]: E0908 19:09:53.821530 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e350895730625185c8ebd16cf5fe34ed41476b4292fa393d1313a5f6f6557a4\": container with ID starting with 3e350895730625185c8ebd16cf5fe34ed41476b4292fa393d1313a5f6f6557a4 not found: ID does not exist" containerID="3e350895730625185c8ebd16cf5fe34ed41476b4292fa393d1313a5f6f6557a4" Sep 08 19:09:53.821654 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.821557 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e350895730625185c8ebd16cf5fe34ed41476b4292fa393d1313a5f6f6557a4"} err="failed to get container status \"3e350895730625185c8ebd16cf5fe34ed41476b4292fa393d1313a5f6f6557a4\": rpc error: code = NotFound desc = could not find container \"3e350895730625185c8ebd16cf5fe34ed41476b4292fa393d1313a5f6f6557a4\": container with ID starting with 3e350895730625185c8ebd16cf5fe34ed41476b4292fa393d1313a5f6f6557a4 not found: ID does not exist" Sep 08 19:09:53.832152 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.832129 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c8c7d87-prjgv"] Sep 08 19:09:53.834270 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:53.834248 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68c8c7d87-prjgv"] Sep 08 19:09:54.591593 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.591556 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68c8c7d87-dvjwh"] Sep 08 19:09:54.591953 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.591878 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d09c6617-3e03-46e8-a313-100512596ca0" containerName="manager" Sep 08 19:09:54.591953 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.591889 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d09c6617-3e03-46e8-a313-100512596ca0" containerName="manager" Sep 08 19:09:54.592026 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.591961 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d09c6617-3e03-46e8-a313-100512596ca0" containerName="manager" Sep 08 19:09:54.596412 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.596389 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8c7d87-dvjwh" Sep 08 19:09:54.599050 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.599012 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-pxxwb\"" Sep 08 19:09:54.599050 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.599027 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 19:09:54.600185 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.600168 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 19:09:54.600317 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.600247 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 19:09:54.606549 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.606529 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c8c7d87-dvjwh"] Sep 08 19:09:54.721653 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.721620 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7hhv\" (UniqueName: \"kubernetes.io/projected/e29233c7-5993-420b-a0e0-4a242faf2041-kube-api-access-d7hhv\") pod \"maas-controller-68c8c7d87-dvjwh\" (UID: \"e29233c7-5993-420b-a0e0-4a242faf2041\") " pod="opendatahub/maas-controller-68c8c7d87-dvjwh" Sep 08 19:09:54.721836 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.721678 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e29233c7-5993-420b-a0e0-4a242faf2041-metrics-certs\") pod \"maas-controller-68c8c7d87-dvjwh\" (UID: \"e29233c7-5993-420b-a0e0-4a242faf2041\") " pod="opendatahub/maas-controller-68c8c7d87-dvjwh" Sep 08 19:09:54.721836 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.721796 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e29233c7-5993-420b-a0e0-4a242faf2041-webhook-cert\") pod \"maas-controller-68c8c7d87-dvjwh\" (UID: \"e29233c7-5993-420b-a0e0-4a242faf2041\") " pod="opendatahub/maas-controller-68c8c7d87-dvjwh" Sep 08 19:09:54.745146 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.745116 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d09c6617-3e03-46e8-a313-100512596ca0" path="/var/lib/kubelet/pods/d09c6617-3e03-46e8-a313-100512596ca0/volumes" Sep 08 19:09:54.823232 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.823206 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e29233c7-5993-420b-a0e0-4a242faf2041-webhook-cert\") pod \"maas-controller-68c8c7d87-dvjwh\" (UID: \"e29233c7-5993-420b-a0e0-4a242faf2041\") " pod="opendatahub/maas-controller-68c8c7d87-dvjwh" Sep 08 19:09:54.823399 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.823335 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7hhv\" (UniqueName: \"kubernetes.io/projected/e29233c7-5993-420b-a0e0-4a242faf2041-kube-api-access-d7hhv\") pod \"maas-controller-68c8c7d87-dvjwh\" (UID: \"e29233c7-5993-420b-a0e0-4a242faf2041\") " pod="opendatahub/maas-controller-68c8c7d87-dvjwh" Sep 08 19:09:54.823810 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.823530 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e29233c7-5993-420b-a0e0-4a242faf2041-metrics-certs\") pod \"maas-controller-68c8c7d87-dvjwh\" (UID: \"e29233c7-5993-420b-a0e0-4a242faf2041\") " pod="opendatahub/maas-controller-68c8c7d87-dvjwh" Sep 08 19:09:54.825830 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.825803 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e29233c7-5993-420b-a0e0-4a242faf2041-webhook-cert\") pod \"maas-controller-68c8c7d87-dvjwh\" (UID: \"e29233c7-5993-420b-a0e0-4a242faf2041\") " pod="opendatahub/maas-controller-68c8c7d87-dvjwh" Sep 08 19:09:54.825950 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.825842 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e29233c7-5993-420b-a0e0-4a242faf2041-metrics-certs\") pod \"maas-controller-68c8c7d87-dvjwh\" (UID: \"e29233c7-5993-420b-a0e0-4a242faf2041\") " pod="opendatahub/maas-controller-68c8c7d87-dvjwh" Sep 08 19:09:54.831999 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.831972 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7hhv\" (UniqueName: \"kubernetes.io/projected/e29233c7-5993-420b-a0e0-4a242faf2041-kube-api-access-d7hhv\") pod \"maas-controller-68c8c7d87-dvjwh\" (UID: \"e29233c7-5993-420b-a0e0-4a242faf2041\") " pod="opendatahub/maas-controller-68c8c7d87-dvjwh" Sep 08 19:09:54.907667 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:54.907640 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c8c7d87-dvjwh" Sep 08 19:09:55.177632 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:55.177606 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c8c7d87-dvjwh"] Sep 08 19:09:55.178930 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:09:55.178901 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29233c7_5993_420b_a0e0_4a242faf2041.slice/crio-a68154d1bf3eae0dfe533f7f1cd06448023179277d6351d4660838e48dc1cdf7 WatchSource:0}: Error finding container a68154d1bf3eae0dfe533f7f1cd06448023179277d6351d4660838e48dc1cdf7: Status 404 returned error can't find the container with id a68154d1bf3eae0dfe533f7f1cd06448023179277d6351d4660838e48dc1cdf7 Sep 08 19:09:55.823065 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:55.823027 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8c7d87-dvjwh" event={"ID":"e29233c7-5993-420b-a0e0-4a242faf2041","Type":"ContainerStarted","Data":"07c2968a6e4d028d144d18ef6df5d75e93ae9dea2dcba0fe356eabd3273adc71"} Sep 08 19:09:55.823439 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:55.823084 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c8c7d87-dvjwh" event={"ID":"e29233c7-5993-420b-a0e0-4a242faf2041","Type":"ContainerStarted","Data":"a68154d1bf3eae0dfe533f7f1cd06448023179277d6351d4660838e48dc1cdf7"} Sep 08 19:09:55.823439 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:55.823236 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68c8c7d87-dvjwh" Sep 08 19:09:55.841422 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:09:55.841315 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68c8c7d87-dvjwh" podStartSLOduration=1.442438155 podStartE2EDuration="1.8413009s" totalImagesPullingTime="398.862745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 19:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:09:55.839844409 +0000 UTC m=+2131.545732644" watchObservedRunningTime="2026-09-08 19:09:55.8413009 +0000 UTC m=+2131.547189131" Sep 08 19:10:06.830296 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:10:06.830262 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68c8c7d87-dvjwh" Sep 08 19:10:25.275257 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:10:25.275173 2553 scope.go:117] "RemoveContainer" containerID="c1cab08259c29b645d6f0475e6c93780fc4bbede6c2063dabc88c46ed1300515" Sep 08 19:10:25.283903 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:10:25.283881 2553 scope.go:117] "RemoveContainer" containerID="28a1d4bdfdd311f6638fd7331566453d0cf3d1676faeeb439b827be446129470" Sep 08 19:10:25.291758 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:10:25.291738 2553 scope.go:117] "RemoveContainer" containerID="5575cfdd6c4d3d910cf8ff6952e28fc79acde7c12013f9e054f6e88fe4da02c2" Sep 08 19:12:25.329554 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:12:25.329522 2553 scope.go:117] "RemoveContainer" containerID="a5108bb8f6b6642b8bc193085a12e06131b54fd682772fa5ba56e148f4ae135b" Sep 08 19:13:25.351182 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:13:25.351105 2553 scope.go:117] "RemoveContainer" containerID="2a0404b902d4d7c46a3ce9593ca77789a3e35e527df813b9c992cc3ca01f6822" Sep 08 19:14:25.055489 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:25.055457 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 19:14:25.059518 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:25.059494 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 19:14:25.063067 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:25.063044 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 19:14:25.066879 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:25.066859 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f4hs5_20d4c356-520e-4dd1-b36d-40197469dd3c/ovn-acl-logging/0.log" Sep 08 19:14:25.371121 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:25.371043 2553 scope.go:117] "RemoveContainer" containerID="9f5215eb89c879e84f144560d8a799f54fd87d366cfa322eb8d795c7fbdd0697" Sep 08 19:14:25.379471 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:25.379453 2553 scope.go:117] "RemoveContainer" containerID="db4f3a9489711bc1260147b08336026740922c4769c45f5e6a2045a1753a265f" Sep 08 19:14:25.387049 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:25.387031 2553 scope.go:117] "RemoveContainer" containerID="e7f9c9a0e49d63c15f2b2853d97ba5a4db1f5c1e028c60dafc5579051b6843ea" Sep 08 19:14:32.814156 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:32.814121 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68c8c7d87-dvjwh_e29233c7-5993-420b-a0e0-4a242faf2041/manager/0.log" Sep 08 19:14:32.925220 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:32.925192 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-m7qsp_c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef/server/0.log" Sep 08 19:14:33.035453 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:33.035424 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wtnf6_603bb783-22ba-4bfb-a0c5-82adf214f386/manager/0.log" Sep 08 19:14:33.145265 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:33.145234 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67894858cf-7tlh5_21ee1433-df0e-4fb6-a807-7286c728c070/copy-manifests/0.log" Sep 08 19:14:33.157977 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:33.157949 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67894858cf-7tlh5_21ee1433-df0e-4fb6-a807-7286c728c070/manager/0.log" Sep 08 19:14:33.916985 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:33.916956 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814900-nbcwm_dff884b1-aad4-4b77-9fbb-5e1d02b41f74/cleanup/0.log" Sep 08 19:14:35.795676 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:35.795647 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z_c29c6702-c2f2-4226-9f37-92680e2bdd62/istio-proxy/0.log" Sep 08 19:14:35.913167 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:35.913134 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-czckm_bc171b62-76f6-462a-8281-b3aa288c422c/discovery/0.log" Sep 08 19:14:36.360545 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:36.360518 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-s6mvc_43765cab-e58f-47fb-9b10-ac65bae9b8e3/payload-pre-processing/0.log" Sep 08 19:14:37.038001 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:37.037958 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-n7l5b_d13b3600-44b7-490a-ba2f-6d96a80be4b5/main/0.log" Sep 08 19:14:37.256468 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:37.256434 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-crx9n_2670c956-bd01-4574-b847-3a1b5c0ed102/main/0.log" Sep 08 19:14:37.484955 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:37.484928 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-2bmkf_6b6cde27-cdeb-47e9-ad27-3e63042fd929/main/0.log" Sep 08 19:14:45.894358 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:45.894320 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tzbd8_f4bc5cda-5c62-4dae-8e79-d6f3f81bf03c/global-pull-secret-syncer/0.log" Sep 08 19:14:45.965616 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:45.965591 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j9kxx_7aa9c205-ae85-4940-ba9a-b9ba08937086/konnectivity-agent/0.log" Sep 08 19:14:46.024211 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:46.024179 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-43.ec2.internal_1c0a1ff402e4c440e263b451c0fbfb11/haproxy/0.log" Sep 08 19:14:52.580014 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:52.579980 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-q5hmp_bda1fd47-da9a-49cd-8f40-b44c70207933/cluster-monitoring-operator/0.log" Sep 08 19:14:52.604443 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:52.604416 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gd5qc_f3594cff-d79f-4fec-97eb-baee86351196/kube-state-metrics/0.log" Sep 08 19:14:52.625340 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:52.625313 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gd5qc_f3594cff-d79f-4fec-97eb-baee86351196/kube-rbac-proxy-main/0.log" Sep 08 19:14:52.646539 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:52.646513 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gd5qc_f3594cff-d79f-4fec-97eb-baee86351196/kube-rbac-proxy-self/0.log" Sep 08 19:14:52.690793 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:52.690760 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-54c88f84f-lpg5p_c3baf95c-aa39-4ecd-8b61-c9a9b4c90bc3/metrics-server/0.log" Sep 08 19:14:52.758468 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:52.758452 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2htzd_c8ede2fd-b04a-4bfe-be85-b74a047eaf5d/node-exporter/0.log" Sep 08 19:14:52.784977 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:52.784958 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2htzd_c8ede2fd-b04a-4bfe-be85-b74a047eaf5d/kube-rbac-proxy/0.log" Sep 08 19:14:52.814271 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:52.814251 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2htzd_c8ede2fd-b04a-4bfe-be85-b74a047eaf5d/init-textfile/0.log" Sep 08 19:14:55.544138 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:55.544104 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/1.log" Sep 08 19:14:55.549852 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:55.549827 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-c6r7f_acc31f4d-9ae3-4be1-8a1d-415cb8fc6041/console-operator/2.log" Sep 08 19:14:55.774636 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:55.774586 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc"] Sep 08 19:14:55.778436 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:55.778415 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:55.781078 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:55.781054 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qgbgk\"/\"openshift-service-ca.crt\"" Sep 08 19:14:55.782134 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:55.782112 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qgbgk\"/\"kube-root-ca.crt\"" Sep 08 19:14:55.782224 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:55.782112 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qgbgk\"/\"default-dockercfg-xsjvx\"" Sep 08 19:14:55.788145 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:55.788114 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc"] Sep 08 19:14:55.935024 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:55.934996 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c00e9fc9-c865-40a7-82a7-0020b2e3b008-lib-modules\") pod \"perf-node-gather-daemonset-gzmxc\" (UID: \"c00e9fc9-c865-40a7-82a7-0020b2e3b008\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:55.935251 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:55.935039 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c00e9fc9-c865-40a7-82a7-0020b2e3b008-proc\") pod \"perf-node-gather-daemonset-gzmxc\" (UID: \"c00e9fc9-c865-40a7-82a7-0020b2e3b008\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:55.935251 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:55.935060 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c00e9fc9-c865-40a7-82a7-0020b2e3b008-podres\") pod \"perf-node-gather-daemonset-gzmxc\" (UID: \"c00e9fc9-c865-40a7-82a7-0020b2e3b008\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:55.935251 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:55.935149 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f58cg\" (UniqueName: \"kubernetes.io/projected/c00e9fc9-c865-40a7-82a7-0020b2e3b008-kube-api-access-f58cg\") pod \"perf-node-gather-daemonset-gzmxc\" (UID: \"c00e9fc9-c865-40a7-82a7-0020b2e3b008\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:55.935251 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:55.935208 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c00e9fc9-c865-40a7-82a7-0020b2e3b008-sys\") pod \"perf-node-gather-daemonset-gzmxc\" (UID: \"c00e9fc9-c865-40a7-82a7-0020b2e3b008\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:56.036391 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:56.036346 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f58cg\" (UniqueName: \"kubernetes.io/projected/c00e9fc9-c865-40a7-82a7-0020b2e3b008-kube-api-access-f58cg\") pod \"perf-node-gather-daemonset-gzmxc\" (UID: \"c00e9fc9-c865-40a7-82a7-0020b2e3b008\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:56.036552 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:56.036414 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c00e9fc9-c865-40a7-82a7-0020b2e3b008-sys\") pod \"perf-node-gather-daemonset-gzmxc\" (UID: \"c00e9fc9-c865-40a7-82a7-0020b2e3b008\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:56.036552 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:56.036447 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c00e9fc9-c865-40a7-82a7-0020b2e3b008-lib-modules\") pod \"perf-node-gather-daemonset-gzmxc\" (UID: \"c00e9fc9-c865-40a7-82a7-0020b2e3b008\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:56.036552 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:56.036486 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c00e9fc9-c865-40a7-82a7-0020b2e3b008-proc\") pod \"perf-node-gather-daemonset-gzmxc\" (UID: \"c00e9fc9-c865-40a7-82a7-0020b2e3b008\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:56.036552 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:56.036522 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c00e9fc9-c865-40a7-82a7-0020b2e3b008-podres\") pod \"perf-node-gather-daemonset-gzmxc\" (UID: \"c00e9fc9-c865-40a7-82a7-0020b2e3b008\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:56.036552 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:56.036538 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c00e9fc9-c865-40a7-82a7-0020b2e3b008-sys\") pod \"perf-node-gather-daemonset-gzmxc\" (UID: \"c00e9fc9-c865-40a7-82a7-0020b2e3b008\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:56.036771 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:56.036589 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c00e9fc9-c865-40a7-82a7-0020b2e3b008-lib-modules\") pod \"perf-node-gather-daemonset-gzmxc\" (UID: \"c00e9fc9-c865-40a7-82a7-0020b2e3b008\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:56.036771 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:56.036632 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c00e9fc9-c865-40a7-82a7-0020b2e3b008-podres\") pod \"perf-node-gather-daemonset-gzmxc\" (UID: \"c00e9fc9-c865-40a7-82a7-0020b2e3b008\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:56.036771 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:56.036630 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c00e9fc9-c865-40a7-82a7-0020b2e3b008-proc\") pod \"perf-node-gather-daemonset-gzmxc\" (UID: \"c00e9fc9-c865-40a7-82a7-0020b2e3b008\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:56.043937 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:56.043918 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f58cg\" (UniqueName: \"kubernetes.io/projected/c00e9fc9-c865-40a7-82a7-0020b2e3b008-kube-api-access-f58cg\") pod \"perf-node-gather-daemonset-gzmxc\" (UID: \"c00e9fc9-c865-40a7-82a7-0020b2e3b008\") " pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:56.089238 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:56.089213 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:56.364573 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:56.364546 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc"] Sep 08 19:14:56.365887 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:14:56.365861 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc00e9fc9_c865_40a7_82a7_0020b2e3b008.slice/crio-cf3abb0326f3c9dff5a568b8c83ec8888ed62ae9f288e4cae0efca657d1d8730 WatchSource:0}: Error finding container cf3abb0326f3c9dff5a568b8c83ec8888ed62ae9f288e4cae0efca657d1d8730: Status 404 returned error can't find the container with id cf3abb0326f3c9dff5a568b8c83ec8888ed62ae9f288e4cae0efca657d1d8730 Sep 08 19:14:56.426006 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:56.425980 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" event={"ID":"c00e9fc9-c865-40a7-82a7-0020b2e3b008","Type":"ContainerStarted","Data":"cf3abb0326f3c9dff5a568b8c83ec8888ed62ae9f288e4cae0efca657d1d8730"} Sep 08 19:14:56.798930 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:56.798901 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-cgjjn_1b6996c8-2d68-429f-a9dd-5b0543a43060/volume-data-source-validator/0.log" Sep 08 19:14:57.432765 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:57.432726 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" event={"ID":"c00e9fc9-c865-40a7-82a7-0020b2e3b008","Type":"ContainerStarted","Data":"7bff97a910ffe3528d33c5562f9322ae9c57e534eca8ab09526a3ceba3d5e72f"} Sep 08 19:14:57.432915 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:57.432787 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:14:57.447669 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:57.447628 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" podStartSLOduration=2.447613927 podStartE2EDuration="2.447613927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 19:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 19:14:57.447165085 +0000 UTC m=+2433.153053318" watchObservedRunningTime="2026-09-08 19:14:57.447613927 +0000 UTC m=+2433.153502159" Sep 08 19:14:57.977539 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:57.977512 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pjc2h_71e540dd-1e1b-46ab-b6d5-01a0ccf6463c/dns/0.log" Sep 08 19:14:57.997330 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:57.997308 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pjc2h_71e540dd-1e1b-46ab-b6d5-01a0ccf6463c/kube-rbac-proxy/0.log" Sep 08 19:14:58.107477 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:58.107447 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xmpj8_96b33d5d-d099-464a-adcc-03562ae049c0/dns-node-resolver/0.log" Sep 08 19:14:58.689243 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:58.689212 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55c57bf558-n9hfq_d3d3761a-ecc1-4045-bf09-80137816248f/registry/0.log" Sep 08 19:14:58.736463 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:58.736436 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jq7qc_1911e404-7356-4fd2-86af-acc9012c5a4b/node-ca/0.log" Sep 08 19:14:59.668575 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:59.668547 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z_c29c6702-c2f2-4226-9f37-92680e2bdd62/istio-proxy/0.log" Sep 08 19:14:59.879874 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:14:59.879843 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-czckm_bc171b62-76f6-462a-8281-b3aa288c422c/discovery/0.log" Sep 08 19:15:00.010473 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:00.010395 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-s6mvc_43765cab-e58f-47fb-9b10-ac65bae9b8e3/payload-pre-processing/0.log" Sep 08 19:15:00.135856 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:00.135823 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814915-f8r6c"] Sep 08 19:15:00.139407 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:00.139381 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814915-f8r6c" Sep 08 19:15:00.142172 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:00.142154 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cttx2\"" Sep 08 19:15:00.147279 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:00.147159 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814915-f8r6c"] Sep 08 19:15:00.150566 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:00.150546 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814915-f8r6c" Sep 08 19:15:00.422429 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:00.422399 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814915-f8r6c"] Sep 08 19:15:00.423708 ip-10-0-132-43 kubenswrapper[2553]: W0908 19:15:00.423678 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460f11af_9920_443c_ac66_2c3da2fe1c73.slice/crio-3084dbd9185f00ed37bcc80a325b9137da9a9164b63644ecc89c67385178cc2e WatchSource:0}: Error finding container 3084dbd9185f00ed37bcc80a325b9137da9a9164b63644ecc89c67385178cc2e: Status 404 returned error can't find the container with id 3084dbd9185f00ed37bcc80a325b9137da9a9164b63644ecc89c67385178cc2e Sep 08 19:15:00.450721 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:00.450693 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814915-f8r6c" event={"ID":"460f11af-9920-443c-ac66-2c3da2fe1c73","Type":"ContainerStarted","Data":"3084dbd9185f00ed37bcc80a325b9137da9a9164b63644ecc89c67385178cc2e"} Sep 08 19:15:00.949450 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:00.949415 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mnblg_2ae8b37f-62ce-4a27-ae4b-369e38c6de40/serve-healthcheck-canary/0.log" Sep 08 19:15:01.373880 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:01.373855 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-phjbj_4fa729ff-ae68-404f-a6c0-a1c87566f68c/insights-operator/0.log" Sep 08 19:15:01.373998 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:01.373935 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-phjbj_4fa729ff-ae68-404f-a6c0-a1c87566f68c/insights-operator/1.log" Sep 08 19:15:01.458686 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:01.458649 2553 generic.go:356] "Generic (PLEG): container finished" podID="460f11af-9920-443c-ac66-2c3da2fe1c73" containerID="045eeb9c802cb02219a2b77365f26a669822fc2b2266b12fc49fcc5348dc2017" exitCode=0 Sep 08 19:15:01.458851 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:01.458692 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814915-f8r6c" event={"ID":"460f11af-9920-443c-ac66-2c3da2fe1c73","Type":"ContainerDied","Data":"045eeb9c802cb02219a2b77365f26a669822fc2b2266b12fc49fcc5348dc2017"} Sep 08 19:15:01.518817 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:01.518788 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nclnr_bcb2c470-bbc4-4b9d-8e81-ad048bb0c693/kube-rbac-proxy/0.log" Sep 08 19:15:01.537092 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:01.537065 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nclnr_bcb2c470-bbc4-4b9d-8e81-ad048bb0c693/exporter/0.log" Sep 08 19:15:01.559437 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:01.559409 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nclnr_bcb2c470-bbc4-4b9d-8e81-ad048bb0c693/extractor/0.log" Sep 08 19:15:02.591028 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:02.590997 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814915-f8r6c" Sep 08 19:15:03.447782 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:03.447753 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qgbgk/perf-node-gather-daemonset-gzmxc" Sep 08 19:15:03.470055 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:03.470029 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814915-f8r6c" Sep 08 19:15:03.470198 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:03.470062 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814915-f8r6c" event={"ID":"460f11af-9920-443c-ac66-2c3da2fe1c73","Type":"ContainerDied","Data":"3084dbd9185f00ed37bcc80a325b9137da9a9164b63644ecc89c67385178cc2e"} Sep 08 19:15:03.470198 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:03.470098 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3084dbd9185f00ed37bcc80a325b9137da9a9164b63644ecc89c67385178cc2e" Sep 08 19:15:04.011146 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:04.011111 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68c8c7d87-dvjwh_e29233c7-5993-420b-a0e0-4a242faf2041/manager/0.log" Sep 08 19:15:04.031302 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:04.031273 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-m7qsp_c4d68d3a-cb6a-4d3a-96b1-3b7a9d162aef/server/0.log" Sep 08 19:15:04.108203 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:04.108168 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wtnf6_603bb783-22ba-4bfb-a0c5-82adf214f386/manager/0.log" Sep 08 19:15:04.192615 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:04.192587 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67894858cf-7tlh5_21ee1433-df0e-4fb6-a807-7286c728c070/manager/0.log" Sep 08 19:15:04.216475 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:04.216443 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67894858cf-7tlh5_21ee1433-df0e-4fb6-a807-7286c728c070/copy-manifests/0.log" Sep 08 19:15:05.738258 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:05.738232 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68bb56fbd7-kw7fb_42e6a3e1-1a51-4ccb-963a-42c6a0afd8d2/manager/0.log" Sep 08 19:15:06.235653 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:06.235623 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z_c29c6702-c2f2-4226-9f37-92680e2bdd62/istio-proxy/0.log" Sep 08 19:15:06.443457 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:06.443347 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-czckm_bc171b62-76f6-462a-8281-b3aa288c422c/discovery/0.log" Sep 08 19:15:06.551952 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:06.551878 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-s6mvc_43765cab-e58f-47fb-9b10-ac65bae9b8e3/payload-pre-processing/0.log" Sep 08 19:15:08.119710 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:08.119678 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rrm2z_c29c6702-c2f2-4226-9f37-92680e2bdd62/istio-proxy/0.log" Sep 08 19:15:08.342661 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:08.342624 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-czckm_bc171b62-76f6-462a-8281-b3aa288c422c/discovery/0.log" Sep 08 19:15:08.449582 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:08.449555 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-s6mvc_43765cab-e58f-47fb-9b10-ac65bae9b8e3/payload-pre-processing/0.log" Sep 08 19:15:11.016233 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:11.016178 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-fbzzv_77e317bc-d870-4bb8-a823-6a240c26a5ff/migrator/0.log" Sep 08 19:15:11.035432 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:11.035396 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-fbzzv_77e317bc-d870-4bb8-a823-6a240c26a5ff/graceful-termination/0.log" Sep 08 19:15:12.488611 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:12.488576 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7pnc2_aa2faa21-f7f1-4010-851b-bd9244d70ea5/kube-multus/0.log" Sep 08 19:15:12.638717 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:12.638686 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gchnh_d8e54f1e-5ce8-481f-a6cb-c01a98251ed7/kube-multus-additional-cni-plugins/0.log" Sep 08 19:15:12.656590 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:12.656564 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gchnh_d8e54f1e-5ce8-481f-a6cb-c01a98251ed7/egress-router-binary-copy/0.log" Sep 08 19:15:12.674532 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:12.674516 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gchnh_d8e54f1e-5ce8-481f-a6cb-c01a98251ed7/cni-plugins/0.log" Sep 08 19:15:12.693055 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:12.693032 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gchnh_d8e54f1e-5ce8-481f-a6cb-c01a98251ed7/bond-cni-plugin/0.log" Sep 08 19:15:12.713196 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:12.713177 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gchnh_d8e54f1e-5ce8-481f-a6cb-c01a98251ed7/routeoverride-cni/0.log" Sep 08 19:15:12.730403 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:12.730383 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gchnh_d8e54f1e-5ce8-481f-a6cb-c01a98251ed7/whereabouts-cni-bincopy/0.log" Sep 08 19:15:12.748756 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:12.748696 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gchnh_d8e54f1e-5ce8-481f-a6cb-c01a98251ed7/whereabouts-cni/0.log" Sep 08 19:15:13.068870 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:13.068788 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sjrbw_40b6e42c-6d96-445d-8b83-108320730adb/network-metrics-daemon/0.log" Sep 08 19:15:13.084307 ip-10-0-132-43 kubenswrapper[2553]: I0908 19:15:13.084282 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-sjrbw_40b6e42c-6d96-445d-8b83-108320730adb/kube-rbac-proxy/0.log"