Sep 11 13:57:39.465767 ip-10-0-136-92 systemd[1]: Starting Kubernetes Kubelet... Sep 11 13:57:39.917096 ip-10-0-136-92 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 13:57:39.917096 ip-10-0-136-92 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 13:57:39.917096 ip-10-0-136-92 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 13:57:39.917096 ip-10-0-136-92 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 13:57:39.917096 ip-10-0-136-92 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 13:57:39.918206 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.918127 2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 13:57:39.922578 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922557 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:57:39.922578 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922576 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:57:39.922578 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922582 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922588 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922591 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922594 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922598 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922600 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922603 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922606 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922610 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922614 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922617 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922620 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922623 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922626 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922629 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922633 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922636 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922639 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922641 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:57:39.922719 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922645 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922647 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922650 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922652 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922655 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922657 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922669 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922671 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922674 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922676 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922679 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922682 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922684 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922687 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922689 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922693 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922696 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922699 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922702 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922705 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:57:39.923180 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922707 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922710 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922712 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922715 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922717 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922720 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922723 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922725 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922728 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922731 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922733 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922736 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922738 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922741 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922743 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922746 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922748 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922750 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922753 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922756 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:57:39.923740 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922759 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922761 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922764 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922768 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922771 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922773 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922776 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922778 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922781 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922784 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922786 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922788 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922791 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922793 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922796 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922798 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922800 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922803 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922805 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:57:39.924236 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922807 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922810 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922812 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922821 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922823 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922826 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922828 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922831 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922833 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922835 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922838 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922840 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922842 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922845 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922847 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922850 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922853 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922855 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.922857 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:57:39.924685 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923595 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923603 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923607 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923611 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923614 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923617 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923620 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923623 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923626 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923629 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923632 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923635 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923637 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923640 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923643 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923645 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923648 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923651 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923654 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923657 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:57:39.925141 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923659 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923662 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923664 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923666 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923669 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923671 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923674 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923676 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923678 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923681 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923684 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923687 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923689 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923692 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923696 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923698 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923701 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923703 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923705 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923708 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:57:39.925665 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923711 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923713 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923716 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923718 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923721 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923723 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923726 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923728 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923730 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923733 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923736 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923739 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923741 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923744 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923746 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923749 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923751 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923754 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923756 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923759 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:57:39.926159 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923761 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923765 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923767 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923770 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923772 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923775 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923777 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923780 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923785 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923788 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923791 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923794 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923797 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923800 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923802 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923805 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923808 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923810 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923813 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:57:39.926627 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923815 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923820 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923823 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923825 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923829 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923832 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923834 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923837 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923839 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923842 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923844 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923846 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923849 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923851 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923854 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923856 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923859 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923861 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923865 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:57:39.927080 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.923870 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924256 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924261 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924264 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924267 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924270 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924272 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924275 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924277 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924280 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924282 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924285 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924287 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924290 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924292 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924295 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924297 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924300 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924303 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924305 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:57:39.927561 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924308 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924310 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924312 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924315 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924317 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924321 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924325 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924328 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924331 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924334 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924337 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924341 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924343 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924346 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924348 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924351 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924353 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924356 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924358 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:57:39.928034 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924361 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924363 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924365 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924368 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924370 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924372 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924376 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924378 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924381 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924383 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924385 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924388 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924391 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924393 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924395 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924398 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924400 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924402 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924405 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924407 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:57:39.928496 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924410 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924413 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924415 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924418 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924421 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924423 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924425 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924428 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924431 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924433 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924435 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924438 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924440 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924443 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924445 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924449 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924452 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924455 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924457 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:57:39.928978 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924460 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924463 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924465 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924469 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924472 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924478 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924482 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924484 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924487 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924489 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924492 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924495 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924497 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924500 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924502 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924505 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924510 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924513 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924515 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924518 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:57:39.929453 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924520 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.924522 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924595 2566 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924632 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924637 2566 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924642 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924646 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924649 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924652 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924657 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924660 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924662 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924666 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924669 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924672 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924675 2566 flags.go:64] FLAG: --cgroup-root="" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924677 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924680 2566 flags.go:64] FLAG: --client-ca-file="" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924687 2566 flags.go:64] FLAG: --cloud-provider="external" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924690 2566 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924695 2566 flags.go:64] FLAG: --cluster-domain="" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924698 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924701 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924704 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 13:57:39.929971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924707 2566 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924710 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924713 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924716 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924721 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924724 2566 flags.go:64] FLAG: --contention-profiling="false" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924727 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924730 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924733 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924736 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924740 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924743 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924745 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924748 2566 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924751 2566 flags.go:64] FLAG: --enable-server="true" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924753 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924767 2566 flags.go:64] FLAG: --event-burst="100" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924770 2566 flags.go:64] FLAG: --event-qps="50" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924773 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924776 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924779 2566 flags.go:64] FLAG: --eviction-hard="" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924782 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924785 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924788 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924791 2566 flags.go:64] FLAG: --eviction-soft="" Sep 11 13:57:39.930547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924793 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924796 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924799 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924802 2566 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924805 2566 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924808 2566 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924811 2566 flags.go:64] FLAG: --feature-gates="" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924814 2566 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924817 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924820 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924823 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924827 2566 flags.go:64] FLAG: --healthz-port="10248" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924830 2566 flags.go:64] FLAG: --help="false" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924833 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-136-92.ec2.internal" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924836 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924838 2566 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924841 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924844 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924847 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924850 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924853 2566 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924855 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924858 2566 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 13:57:39.931158 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924861 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924863 2566 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924866 2566 flags.go:64] FLAG: --kube-reserved="" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924869 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924871 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924874 2566 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924877 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924879 2566 flags.go:64] FLAG: --lock-file="" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924882 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924885 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924887 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924892 2566 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924897 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924900 2566 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924902 2566 flags.go:64] FLAG: --logging-format="text" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924905 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924908 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924911 2566 flags.go:64] FLAG: --manifest-url="" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924914 2566 flags.go:64] FLAG: --manifest-url-header="" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924918 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924922 2566 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924930 2566 flags.go:64] FLAG: --max-pods="110" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924933 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924936 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924938 2566 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 13:57:39.931752 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924941 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924944 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924947 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924949 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924957 2566 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924959 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924962 2566 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924965 2566 flags.go:64] FLAG: --pod-cidr="" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924968 2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924973 2566 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924976 2566 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924978 2566 flags.go:64] FLAG: --pods-per-core="0" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924981 2566 flags.go:64] FLAG: --port="10250" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924984 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924987 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0880c747ddea2a19b" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924990 2566 flags.go:64] FLAG: --qos-reserved="" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924992 2566 flags.go:64] FLAG: --read-only-port="10255" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924995 2566 flags.go:64] FLAG: --register-node="true" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.924998 2566 flags.go:64] FLAG: --register-with-taints="" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925001 2566 flags.go:64] FLAG: --registry-burst="10" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925004 2566 flags.go:64] FLAG: --registry-qps="5" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925007 2566 flags.go:64] FLAG: --reserved-cpus="" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925010 2566 flags.go:64] FLAG: --reserved-memory="" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925013 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925016 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 13:57:39.932372 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925019 2566 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925022 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925027 2566 flags.go:64] FLAG: --runonce="false" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925030 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925032 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925035 2566 flags.go:64] FLAG: --seccomp-default="false" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925038 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925041 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925044 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925047 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925049 2566 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925052 2566 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925055 2566 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925058 2566 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925061 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925064 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925066 2566 flags.go:64] FLAG: --system-cgroups="" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925069 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925074 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925077 2566 flags.go:64] FLAG: --tls-cert-file="" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925080 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925099 2566 flags.go:64] FLAG: --tls-min-version="" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925102 2566 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925104 2566 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925107 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 13:57:39.932961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925110 2566 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925112 2566 flags.go:64] FLAG: --v="2" Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925116 2566 flags.go:64] FLAG: --version="false" Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925121 2566 flags.go:64] FLAG: --vmodule="" Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925124 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.925130 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925242 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925246 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925249 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925253 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925256 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925259 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925262 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925265 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925268 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925270 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925273 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925275 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925278 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925280 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925283 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:57:39.933620 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925285 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925288 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925290 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925293 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925295 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925298 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925300 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925303 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925305 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925308 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925311 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925313 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925315 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925318 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925320 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925329 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925333 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925336 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925338 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925341 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:57:39.934153 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925344 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925347 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925349 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925352 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925354 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925357 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925359 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925362 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925364 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925367 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925369 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925371 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925374 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925376 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925379 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925381 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925384 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925387 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925389 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925392 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:57:39.934626 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925394 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925397 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925399 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925401 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925404 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925406 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925409 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925411 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925418 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925420 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925423 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925427 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925431 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925434 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925437 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925439 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925442 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925445 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925447 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:57:39.935137 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925450 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925452 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925455 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925457 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925460 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925463 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925465 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925468 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925470 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925474 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925477 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925480 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925482 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925485 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925488 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925492 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925496 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925500 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925503 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925506 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:57:39.935607 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925508 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:57:39.936152 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925512 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:57:39.936152 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925516 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:57:39.936152 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925519 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:57:39.936152 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.925523 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:57:39.936152 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.926186 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 13:57:39.936152 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.933736 2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 13:57:39.936152 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.933751 2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 13:57:39.936152 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.934566 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 13:57:39.936152 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.934574 2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 13:57:39.936152 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934630 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:57:39.936152 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934636 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:57:39.936152 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934639 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:57:39.936152 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934642 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:57:39.936152 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934645 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934648 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934650 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934653 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934655 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934658 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934660 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934663 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934666 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934669 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934671 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934674 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934677 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934679 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934682 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934684 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934687 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934690 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934692 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934695 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:57:39.936506 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934697 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934699 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934702 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934705 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934707 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934712 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934715 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934717 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934720 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934722 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934725 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934727 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934730 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934734 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934737 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934740 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934743 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934746 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934749 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:57:39.937015 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934752 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934754 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934757 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934760 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934763 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934765 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934768 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934770 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934773 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934775 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934778 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934780 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934783 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934785 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934788 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934790 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934793 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934796 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934799 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934802 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:57:39.937491 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934804 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934807 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934810 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934812 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934814 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934817 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934819 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934822 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934824 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934826 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934829 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934831 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934834 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934836 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934839 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934841 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934844 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934846 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934849 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934853 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:57:39.937965 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934855 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:57:39.938460 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934858 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:57:39.938460 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934861 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:57:39.938460 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934863 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:57:39.938460 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934866 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:57:39.938460 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934869 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:57:39.938460 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934871 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:57:39.938460 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934874 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:57:39.938460 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934877 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:57:39.938460 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934879 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:57:39.938460 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934882 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:57:39.938460 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934886 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:57:39.938460 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934890 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:57:39.938460 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934893 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:57:39.938460 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934895 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:57:39.938460 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.934898 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:57:39.938825 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.934903 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 13:57:39.938825 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935015 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:57:39.938825 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935020 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:57:39.938825 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935023 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:57:39.938825 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935025 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:57:39.938825 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935028 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:57:39.938825 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935030 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:57:39.938825 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935033 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:57:39.938825 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935035 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:57:39.938825 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935038 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:57:39.938825 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935040 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:57:39.938825 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935044 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:57:39.938825 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935048 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:57:39.938825 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935052 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:57:39.938825 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935055 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935057 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935060 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935063 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935065 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935068 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935070 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935072 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935075 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935078 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935097 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935103 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935106 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935108 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935111 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935119 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935121 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935124 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935127 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935129 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:57:39.939211 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935132 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935134 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935137 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935139 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935142 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935145 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935148 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935150 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935153 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935155 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935158 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935160 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935163 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935166 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935168 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935171 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935173 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935176 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935178 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935181 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:57:39.939695 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935183 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935186 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935189 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935193 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935196 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935198 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935202 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935204 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935207 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935210 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935213 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935215 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935218 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935221 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935223 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935225 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935228 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935230 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935234 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:57:39.940191 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935237 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935240 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935243 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935245 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935247 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935250 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935252 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935254 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935257 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935260 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935262 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935264 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935267 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935269 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935272 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935274 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935277 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935279 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935282 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935285 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:57:39.940646 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935287 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:57:39.941127 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935290 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:57:39.941127 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935292 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:57:39.941127 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935295 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:57:39.941127 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935297 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:57:39.941127 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935300 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:57:39.941127 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:39.935302 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:57:39.941127 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.935307 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 13:57:39.941127 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.935417 2566 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 13:57:39.941127 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.937751 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 13:57:39.941127 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.938770 2566 server.go:1019] "Starting client certificate rotation" Sep 11 13:57:39.941127 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.938871 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 13:57:39.941127 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.938909 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 13:57:39.941747 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.941733 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 13:57:39.942896 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.942876 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 13:57:39.956872 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.956846 2566 log.go:25] "Validated CRI v1 runtime API" Sep 11 13:57:39.961540 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.961516 2566 log.go:25] "Validated CRI v1 image API" Sep 11 13:57:39.962450 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.962428 2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 13:57:39.966455 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.966438 2566 fs.go:135] Filesystem UUIDs: map[341009e7-202d-4358-bd28-f5a83400e991:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7c221596-fb2c-4975-9184-005150c4556b:/dev/nvme0n1p3] Sep 11 13:57:39.966502 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.966456 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 13:57:39.972702 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.972574 2566 manager.go:216] Machine: {Timestamp:2026-09-11 13:57:39.971171894 +0000 UTC m=+0.376153013 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101375 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d260fec5030050e67ff7a13d0a048 SystemUUID:ec2d260f-ec50-3005-0e67-ff7a13d0a048 BootID:2781ed67-3fc7-40ab-8d1e-0224fcf4d7a2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3d:91:50:58:9b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3d:91:50:58:9b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:ae:88:0e:ed:d1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 13:57:39.972702 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.972693 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 13:57:39.972831 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.972779 2566 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 13:57:39.975441 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.975423 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 13:57:39.976483 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.976457 2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 13:57:39.976636 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.976489 2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-92.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 13:57:39.977398 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.977385 2566 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 13:57:39.977465 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.977403 2566 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 13:57:39.977465 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.977413 2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 13:57:39.977465 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.977431 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 13:57:39.978675 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.978663 2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 13:57:39.982364 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.982350 2566 state_mem.go:36] "Initialized new in-memory state store" Sep 11 13:57:39.982495 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.982484 2566 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 13:57:39.982567 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.982521 2566 kubelet.go:486] "Attempting to sync node with API server" Sep 11 13:57:39.982567 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.982538 2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 13:57:39.982567 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.982558 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 13:57:39.982683 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.982571 2566 kubelet.go:398] "Adding apiserver pod source" Sep 11 13:57:39.982683 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.982585 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 13:57:39.982941 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.982928 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 13:57:39.983000 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.982951 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 13:57:39.985886 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.985871 2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 13:57:39.988536 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.988522 2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 13:57:39.988597 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.988553 2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 13:57:39.990481 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.990459 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 13:57:39.990481 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.990477 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 13:57:39.990588 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.990493 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 13:57:39.990588 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.990510 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 13:57:39.990588 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.990522 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 13:57:39.990588 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.990532 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 13:57:39.990588 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.990550 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 13:57:39.990588 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.990558 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 13:57:39.990588 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.990567 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 13:57:39.990588 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.990576 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 13:57:39.990810 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.990601 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 13:57:39.990810 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.990610 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 13:57:39.991753 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.991740 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 13:57:39.991753 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.991754 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 13:57:39.992824 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:39.992798 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 13:57:39.992867 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:39.992806 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 13:57:39.995652 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.995640 2566 server.go:1268] "Started kubelet" Sep 11 13:57:39.995755 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.995729 2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 13:57:39.995916 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:39.995713 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 13:57:40.000262 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.000234 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 13:57:40.000789 ip-10-0-136-92 systemd[1]: Started Kubernetes Kubelet. Sep 11 13:57:40.000901 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.000476 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6plbx" Sep 11 13:57:40.003187 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.002155 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-92.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 13:57:40.003187 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.002639 2566 server.go:310] "Adding debug handlers to kubelet server" Sep 11 13:57:40.003187 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.002823 2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 13:57:40.003497 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.003467 2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 13:57:40.008240 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.007182 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-92.ec2.internal.18d448db9a3c5667 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-92.ec2.internal,UID:ip-10-0-136-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-92.ec2.internal,},FirstTimestamp:2026-09-11 13:57:39.995612775 +0000 UTC m=+0.400593894,LastTimestamp:2026-09-11 13:57:39.995612775 +0000 UTC m=+0.400593894,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-92.ec2.internal,}" Sep 11 13:57:40.010021 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.010004 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6plbx" Sep 11 13:57:40.011527 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.011509 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 13:57:40.012142 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.012123 2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 13:57:40.012309 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.012292 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-92.ec2.internal\" not found" Sep 11 13:57:40.012823 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.012806 2566 factory.go:55] Registering systemd factory Sep 11 13:57:40.012909 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.012827 2566 factory.go:223] Registration of the systemd container factory successfully Sep 11 13:57:40.012955 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.012923 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 13:57:40.012955 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.012932 2566 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 13:57:40.012955 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.012946 2566 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 13:57:40.013098 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.013019 2566 factory.go:153] Registering CRI-O factory Sep 11 13:57:40.013098 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.013030 2566 factory.go:223] Registration of the crio container factory successfully Sep 11 13:57:40.013098 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.013047 2566 reconstruct.go:97] "Volume reconstruction finished" Sep 11 13:57:40.013098 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.013053 2566 reconciler.go:29] "Reconciler: start to sync state" Sep 11 13:57:40.013098 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.013073 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 13:57:40.013295 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.013106 2566 factory.go:103] Registering Raw factory Sep 11 13:57:40.013295 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.013120 2566 manager.go:1196] Started watching for new ooms in manager Sep 11 13:57:40.013593 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.013522 2566 manager.go:349] Starting recovery of all containers Sep 11 13:57:40.014557 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.014527 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:57:40.018383 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.018357 2566 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-92.ec2.internal\" not found" node="ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.023734 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.023592 2566 manager.go:354] Recovery completed Sep 11 13:57:40.028056 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.028045 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:57:40.034986 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.034966 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:57:40.035053 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.035003 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:57:40.035053 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.035014 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:57:40.035640 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.035626 2566 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 13:57:40.035640 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.035634 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:57:40.035739 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.035640 2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 13:57:40.035739 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.035665 2566 state_mem.go:36] "Initialized new in-memory state store" Sep 11 13:57:40.036447 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.036432 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:57:40.036511 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.036458 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:57:40.036511 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.036468 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:57:40.038361 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.038348 2566 policy_none.go:49] "None policy: Start" Sep 11 13:57:40.038413 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.038368 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 13:57:40.038413 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.038382 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 13:57:40.039281 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.039269 2566 policy_none.go:47] "Start" Sep 11 13:57:40.081806 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.081778 2566 manager.go:337] "Starting Device Plugin manager" Sep 11 13:57:40.090515 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.081828 2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 13:57:40.090515 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.081843 2566 server.go:87] "Starting device plugin registration server" Sep 11 13:57:40.090515 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.082185 2566 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 13:57:40.090515 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.082196 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 13:57:40.090515 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.082355 2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 13:57:40.090515 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.082488 2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 13:57:40.090515 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.082499 2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 13:57:40.090515 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.082987 2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 13:57:40.090515 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.083044 2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-92.ec2.internal\" not found" Sep 11 13:57:40.090515 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.089421 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 13:57:40.090819 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.090678 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 13:57:40.090819 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.090721 2566 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 13:57:40.090819 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.090740 2566 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 13:57:40.090819 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.090812 2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 13:57:40.093402 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.093385 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:57:40.183377 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.183242 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:57:40.184394 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.184374 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:57:40.184515 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.184409 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:57:40.184515 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.184422 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:57:40.184515 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.184453 2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.191548 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.191530 2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-92.ec2.internal"] Sep 11 13:57:40.191605 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.191597 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:57:40.193219 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.193203 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:57:40.193291 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.193230 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:57:40.193291 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.193238 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:57:40.194592 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.194581 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:57:40.194744 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.194728 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.194813 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.194764 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:57:40.195342 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.195324 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:57:40.195423 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.195355 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:57:40.195423 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.195373 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:57:40.195423 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.195384 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:57:40.195423 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.195358 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:57:40.195630 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.195443 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:57:40.196558 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.196538 2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.196656 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.196565 2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-92.ec2.internal\": node \"ip-10-0-136-92.ec2.internal\" not found" Sep 11 13:57:40.196732 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.196717 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.196768 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.196750 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:57:40.197564 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.197546 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:57:40.197650 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.197578 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:57:40.197650 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.197594 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:57:40.214872 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.214849 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f35c18b2576be95c1be41030bb842c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal\" (UID: \"4f35c18b2576be95c1be41030bb842c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.214963 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.214875 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f35c18b2576be95c1be41030bb842c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal\" (UID: \"4f35c18b2576be95c1be41030bb842c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.214963 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.214893 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/504a2a5e8c9cf487f28426e21025e719-config\") pod \"kube-apiserver-proxy-ip-10-0-136-92.ec2.internal\" (UID: \"504a2a5e8c9cf487f28426e21025e719\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.219796 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.219781 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-92.ec2.internal\" not found" node="ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.224040 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.224024 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-92.ec2.internal\" not found" node="ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.251638 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.251615 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-92.ec2.internal\" not found" Sep 11 13:57:40.315354 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.315334 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f35c18b2576be95c1be41030bb842c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal\" (UID: \"4f35c18b2576be95c1be41030bb842c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.315462 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.315361 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f35c18b2576be95c1be41030bb842c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal\" (UID: \"4f35c18b2576be95c1be41030bb842c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.315462 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.315377 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/504a2a5e8c9cf487f28426e21025e719-config\") pod \"kube-apiserver-proxy-ip-10-0-136-92.ec2.internal\" (UID: \"504a2a5e8c9cf487f28426e21025e719\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.315462 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.315437 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/504a2a5e8c9cf487f28426e21025e719-config\") pod \"kube-apiserver-proxy-ip-10-0-136-92.ec2.internal\" (UID: \"504a2a5e8c9cf487f28426e21025e719\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.315568 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.315437 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f35c18b2576be95c1be41030bb842c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal\" (UID: \"4f35c18b2576be95c1be41030bb842c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.315568 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.315465 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f35c18b2576be95c1be41030bb842c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal\" (UID: \"4f35c18b2576be95c1be41030bb842c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.352444 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.352418 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-92.ec2.internal\" not found" Sep 11 13:57:40.453183 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.453126 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-92.ec2.internal\" not found" Sep 11 13:57:40.523595 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.523564 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.528705 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.528688 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.553533 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.553495 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-92.ec2.internal\" not found" Sep 11 13:57:40.654059 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.654022 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-92.ec2.internal\" not found" Sep 11 13:57:40.754580 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.754508 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-92.ec2.internal\" not found" Sep 11 13:57:40.850032 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.850000 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:57:40.912553 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.912521 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.927975 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.927951 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 13:57:40.929942 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.929912 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.939070 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.939043 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 13:57:40.939279 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.939252 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 13:57:40.939279 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.939253 2566 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://ae35bdc26d50c4087ab92c6aeb0d9fd9-1bfb543498d20572.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.136.92:53332->54.157.203.7:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal" Sep 11 13:57:40.939392 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.939261 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 13:57:40.939392 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.939335 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 13:57:40.962055 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.962024 2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:57:40.983176 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.983150 2566 apiserver.go:52] "Watching apiserver" Sep 11 13:57:40.989293 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.989275 2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 13:57:40.992943 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.991517 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-4wl44","kube-system/konnectivity-agent-zvnz6","kube-system/kube-apiserver-proxy-ip-10-0-136-92.ec2.internal","openshift-dns/node-resolver-pm6vt","openshift-multus/multus-additional-cni-plugins-crfj6","openshift-multus/network-metrics-daemon-hphpv","openshift-network-diagnostics/network-check-target-snsqs","openshift-network-operator/iptables-alerter-xhtdn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn","openshift-cluster-node-tuning-operator/tuned-pcwfb","openshift-image-registry/node-ca-mjmwj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal","openshift-multus/multus-rtct4","openshift-ovn-kubernetes/ovnkube-node-t6zm9"] Sep 11 13:57:40.993180 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.993130 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:40.993608 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.993576 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:57:40.994131 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.994112 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rtct4" Sep 11 13:57:40.995450 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.995428 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:40.996464 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.996447 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pm6vt" Sep 11 13:57:40.996827 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.996798 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 13:57:40.996939 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.996852 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 13:57:40.996939 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.996863 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 13:57:40.996939 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.996914 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 13:57:40.997596 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.997575 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mr5fm\"" Sep 11 13:57:40.997680 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.997580 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:40.998366 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.998351 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 13:57:40.998660 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.998645 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:40.998767 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:40.998699 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:57:40.998833 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.998808 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 13:57:40.998922 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.998870 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 13:57:40.999050 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.999037 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 13:57:40.999119 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.999048 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 13:57:41.000020 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.999742 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.000020 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.999779 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 13:57:41.000020 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.999788 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d7p27\"" Sep 11 13:57:41.000020 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.999786 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 13:57:41.000020 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.999854 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 13:57:41.000020 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.999933 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lp7gp\"" Sep 11 13:57:41.000020 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:40.999944 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qj52f\"" Sep 11 13:57:41.000393 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.000030 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 13:57:41.000457 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.000415 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 13:57:41.001170 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.001156 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:41.001217 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.001203 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:57:41.002238 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.002225 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zvnz6" Sep 11 13:57:41.003667 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.003651 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.004896 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.004848 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mjmwj" Sep 11 13:57:41.005521 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.005505 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 13:57:41.006017 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.006001 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xhtdn" Sep 11 13:57:41.006875 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.006863 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 13:57:41.009420 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.009398 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ltq2c\"" Sep 11 13:57:41.009647 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.009623 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 13:57:41.009750 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.009733 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 13:57:41.009809 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.009748 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 13:57:41.009983 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.009967 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 13:57:41.010202 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.010186 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 13:57:41.010277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.010203 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 13:57:41.010277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.010240 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 13:57:41.010277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.010194 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-skslg\"" Sep 11 13:57:41.010277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.010262 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 13:57:41.011102 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.010454 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 13:57:41.011102 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.010516 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qn4z6\"" Sep 11 13:57:41.011102 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.010545 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s7ltf\"" Sep 11 13:57:41.011102 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.010583 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 13:57:41.011102 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.010601 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kwkxt\"" Sep 11 13:57:41.011102 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.010651 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 13:57:41.011424 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.011332 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 13:52:40 +0000 UTC" deadline="2028-04-19 00:09:30.428628336 +0000 UTC" Sep 11 13:57:41.011424 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.011377 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14050h11m49.417253459s" Sep 11 13:57:41.012362 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.012348 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 13:57:41.014011 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.013997 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 13:57:41.019563 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.019546 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz4q6\" (UniqueName: \"kubernetes.io/projected/681bd531-3870-455f-bde7-39c25302c551-kube-api-access-kz4q6\") pod \"iptables-alerter-xhtdn\" (UID: \"681bd531-3870-455f-bde7-39c25302c551\") " pod="openshift-network-operator/iptables-alerter-xhtdn" Sep 11 13:57:41.019651 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.019572 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/94ee8113-90df-4db6-86d6-f042e909e9c7-ovn-node-metrics-cert\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.019651 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.019596 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/94ee8113-90df-4db6-86d6-f042e909e9c7-ovnkube-script-lib\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.019651 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.019621 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49578359-6ee4-4d62-94ac-dd9b801007e0-cni-binary-copy\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.019765 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.019669 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-kubelet\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.019765 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.019700 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-var-lib-openvswitch\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.019765 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.019724 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cngjm\" (UniqueName: \"kubernetes.io/projected/0ffa4002-eab4-4d53-aad1-c82ba3810441-kube-api-access-cngjm\") pod \"node-resolver-pm6vt\" (UID: \"0ffa4002-eab4-4d53-aad1-c82ba3810441\") " pod="openshift-dns/node-resolver-pm6vt" Sep 11 13:57:41.019888 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.019762 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49578359-6ee4-4d62-94ac-dd9b801007e0-system-cni-dir\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.019888 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.019786 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-cnibin\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.019888 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.019809 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-var-lib-cni-bin\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.020008 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.019840 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-run-ovn\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.020161 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020139 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-device-dir\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.020263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020182 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4krc\" (UniqueName: \"kubernetes.io/projected/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-kube-api-access-f4krc\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.020263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020213 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-multus-conf-dir\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.020263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020242 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-multus-daemon-config\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.020417 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020283 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-run-systemd\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.020417 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020343 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-node-log\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.020417 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020375 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jxqf\" (UniqueName: \"kubernetes.io/projected/94ee8113-90df-4db6-86d6-f042e909e9c7-kube-api-access-9jxqf\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.020417 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020402 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49578359-6ee4-4d62-94ac-dd9b801007e0-cnibin\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.020597 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020430 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2c22dff-0209-4bda-9a7a-5626d2268974-host\") pod \"node-ca-mjmwj\" (UID: \"c2c22dff-0209-4bda-9a7a-5626d2268974\") " pod="openshift-image-registry/node-ca-mjmwj" Sep 11 13:57:41.020597 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020489 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2c22dff-0209-4bda-9a7a-5626d2268974-serviceca\") pod \"node-ca-mjmwj\" (UID: \"c2c22dff-0209-4bda-9a7a-5626d2268974\") " pod="openshift-image-registry/node-ca-mjmwj" Sep 11 13:57:41.020597 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020524 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-run-multus-certs\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.020597 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020555 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.020597 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020591 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49578359-6ee4-4d62-94ac-dd9b801007e0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.020802 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020620 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ec98f86-dcb9-42cf-9add-e2045fbbe773-agent-certs\") pod \"konnectivity-agent-zvnz6\" (UID: \"9ec98f86-dcb9-42cf-9add-e2045fbbe773\") " pod="kube-system/konnectivity-agent-zvnz6" Sep 11 13:57:41.020802 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020656 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-sysctl-conf\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.020802 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020695 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-sys\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.020802 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020739 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-host\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.020802 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020782 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-run-openvswitch\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.021011 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020828 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-log-socket\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.021011 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020852 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-run-k8s-cni-cncf-io\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.021011 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020891 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-system-cni-dir\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.021011 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020920 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-multus-socket-dir-parent\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.021011 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020942 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr9ks\" (UniqueName: \"kubernetes.io/projected/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-kube-api-access-dr9ks\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.021011 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.020970 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-dbus\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:41.021297 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021014 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-sysctl-d\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.021297 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021040 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-systemd\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.021297 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021067 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8t5m\" (UniqueName: \"kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m\") pod \"network-check-target-snsqs\" (UID: \"ce87902b-b413-4c11-8704-f874a61e1f83\") " pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:41.021297 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021106 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-var-lib-kubelet\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.021297 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021136 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-etc-openvswitch\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.021297 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021163 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/94ee8113-90df-4db6-86d6-f042e909e9c7-ovnkube-config\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.021297 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021189 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/94ee8113-90df-4db6-86d6-f042e909e9c7-env-overrides\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.021297 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021218 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49578359-6ee4-4d62-94ac-dd9b801007e0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.021638 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021294 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-etc-selinux\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.021638 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021333 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-modprobe-d\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.021638 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021363 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/681bd531-3870-455f-bde7-39c25302c551-host-slash\") pod \"iptables-alerter-xhtdn\" (UID: \"681bd531-3870-455f-bde7-39c25302c551\") " pod="openshift-network-operator/iptables-alerter-xhtdn" Sep 11 13:57:41.021638 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021386 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-socket-dir\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.021638 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021414 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-var-lib-kubelet\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.021638 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021447 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-var-lib-cni-multus\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.021638 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021476 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49578359-6ee4-4d62-94ac-dd9b801007e0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.021638 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021505 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:41.021638 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021527 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-slash\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.021638 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021554 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/681bd531-3870-455f-bde7-39c25302c551-iptables-alerter-script\") pod \"iptables-alerter-xhtdn\" (UID: \"681bd531-3870-455f-bde7-39c25302c551\") " pod="openshift-network-operator/iptables-alerter-xhtdn" Sep 11 13:57:41.021638 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021581 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7kg2\" (UniqueName: \"kubernetes.io/projected/c2c22dff-0209-4bda-9a7a-5626d2268974-kube-api-access-t7kg2\") pod \"node-ca-mjmwj\" (UID: \"c2c22dff-0209-4bda-9a7a-5626d2268974\") " pod="openshift-image-registry/node-ca-mjmwj" Sep 11 13:57:41.021638 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021607 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-sysconfig\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.021638 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021634 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-etc-kubernetes\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021661 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-cni-netd\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021759 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ffa4002-eab4-4d53-aad1-c82ba3810441-hosts-file\") pod \"node-resolver-pm6vt\" (UID: \"0ffa4002-eab4-4d53-aad1-c82ba3810441\") " pod="openshift-dns/node-resolver-pm6vt" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021788 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-registration-dir\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021812 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-run-netns\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021839 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-systemd-units\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021870 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-run-netns\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021898 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-kubelet-config\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021926 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275zb\" (UniqueName: \"kubernetes.io/projected/98e24790-f943-4aa7-b079-b8940fb02a82-kube-api-access-275zb\") pod \"network-metrics-daemon-hphpv\" (UID: \"98e24790-f943-4aa7-b079-b8940fb02a82\") " pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021955 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-kubernetes\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.021982 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-lib-modules\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.022004 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-cni-binary-copy\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.022036 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-hostroot\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.022067 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.022117 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ffa4002-eab4-4d53-aad1-c82ba3810441-tmp-dir\") pod \"node-resolver-pm6vt\" (UID: \"0ffa4002-eab4-4d53-aad1-c82ba3810441\") " pod="openshift-dns/node-resolver-pm6vt" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.022146 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.022277 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.022188 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs\") pod \"network-metrics-daemon-hphpv\" (UID: \"98e24790-f943-4aa7-b079-b8940fb02a82\") " pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:41.023119 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.022222 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-tuned\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.023119 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.022250 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7673f829-d14f-4abb-8c72-106d9dfd0a20-tmp\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.023119 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.022279 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-multus-cni-dir\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.023119 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.022325 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-cni-bin\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.023119 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.022355 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkmlh\" (UniqueName: \"kubernetes.io/projected/49578359-6ee4-4d62-94ac-dd9b801007e0-kube-api-access-hkmlh\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.023119 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.022383 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl8kl\" (UniqueName: \"kubernetes.io/projected/7673f829-d14f-4abb-8c72-106d9dfd0a20-kube-api-access-sl8kl\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.023119 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.022411 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-sys-fs\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.023119 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.022437 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ec98f86-dcb9-42cf-9add-e2045fbbe773-konnectivity-ca\") pod \"konnectivity-agent-zvnz6\" (UID: \"9ec98f86-dcb9-42cf-9add-e2045fbbe773\") " pod="kube-system/konnectivity-agent-zvnz6" Sep 11 13:57:41.023119 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.022460 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-run\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.026611 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:41.026567 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f35c18b2576be95c1be41030bb842c9.slice/crio-1d91994dd82a10068496ca75ea99214f87da93fa785a158c0b6cbbe05dbd3935 WatchSource:0}: Error finding container 1d91994dd82a10068496ca75ea99214f87da93fa785a158c0b6cbbe05dbd3935: Status 404 returned error can't find the container with id 1d91994dd82a10068496ca75ea99214f87da93fa785a158c0b6cbbe05dbd3935 Sep 11 13:57:41.029971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.029945 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 13:57:41.031416 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.031396 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 13:57:41.062109 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.062073 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sg4rf" Sep 11 13:57:41.078946 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.078924 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sg4rf" Sep 11 13:57:41.093783 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.093747 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal" event={"ID":"4f35c18b2576be95c1be41030bb842c9","Type":"ContainerStarted","Data":"1d91994dd82a10068496ca75ea99214f87da93fa785a158c0b6cbbe05dbd3935"} Sep 11 13:57:41.123028 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123000 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-sys\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.123169 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123026 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-host\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.123169 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123067 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-run-openvswitch\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.123169 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123100 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-log-socket\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.123169 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123120 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-run-k8s-cni-cncf-io\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.123169 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123134 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-sys\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.123169 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123151 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-host\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.123169 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123162 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-log-socket\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123157 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-run-k8s-cni-cncf-io\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123152 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-run-openvswitch\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123198 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-system-cni-dir\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123221 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-multus-socket-dir-parent\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123237 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr9ks\" (UniqueName: \"kubernetes.io/projected/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-kube-api-access-dr9ks\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123256 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-dbus\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123280 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-sysctl-d\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123288 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-system-cni-dir\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123295 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-multus-socket-dir-parent\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123303 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-systemd\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123345 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-systemd\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123362 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8t5m\" (UniqueName: \"kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m\") pod \"network-check-target-snsqs\" (UID: \"ce87902b-b413-4c11-8704-f874a61e1f83\") " pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123362 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-dbus\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123369 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-sysctl-d\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123384 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-var-lib-kubelet\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123433 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-etc-openvswitch\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123457 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/94ee8113-90df-4db6-86d6-f042e909e9c7-ovnkube-config\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.123506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123497 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/94ee8113-90df-4db6-86d6-f042e909e9c7-env-overrides\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123503 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-etc-openvswitch\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123468 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-var-lib-kubelet\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123528 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49578359-6ee4-4d62-94ac-dd9b801007e0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123599 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-etc-selinux\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123627 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-modprobe-d\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123651 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/681bd531-3870-455f-bde7-39c25302c551-host-slash\") pod \"iptables-alerter-xhtdn\" (UID: \"681bd531-3870-455f-bde7-39c25302c551\") " pod="openshift-network-operator/iptables-alerter-xhtdn" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123674 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-socket-dir\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123687 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-etc-selinux\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123715 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-var-lib-kubelet\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123716 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/49578359-6ee4-4d62-94ac-dd9b801007e0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123744 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-modprobe-d\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123744 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-var-lib-cni-multus\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123783 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-var-lib-cni-multus\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123756 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/681bd531-3870-455f-bde7-39c25302c551-host-slash\") pod \"iptables-alerter-xhtdn\" (UID: \"681bd531-3870-455f-bde7-39c25302c551\") " pod="openshift-network-operator/iptables-alerter-xhtdn" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123785 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-var-lib-kubelet\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123785 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49578359-6ee4-4d62-94ac-dd9b801007e0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.124244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123823 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123848 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-slash\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123872 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/681bd531-3870-455f-bde7-39c25302c551-iptables-alerter-script\") pod \"iptables-alerter-xhtdn\" (UID: \"681bd531-3870-455f-bde7-39c25302c551\") " pod="openshift-network-operator/iptables-alerter-xhtdn" Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123899 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7kg2\" (UniqueName: \"kubernetes.io/projected/c2c22dff-0209-4bda-9a7a-5626d2268974-kube-api-access-t7kg2\") pod \"node-ca-mjmwj\" (UID: \"c2c22dff-0209-4bda-9a7a-5626d2268974\") " pod="openshift-image-registry/node-ca-mjmwj" Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123934 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/94ee8113-90df-4db6-86d6-f042e909e9c7-env-overrides\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123949 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-slash\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.123825 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-socket-dir\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.123982 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124052 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/94ee8113-90df-4db6-86d6-f042e909e9c7-ovnkube-config\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.124060 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret podName:3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:41.624040103 +0000 UTC m=+2.029021217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret") pod "global-pull-secret-syncer-4wl44" (UID: "3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124380 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/49578359-6ee4-4d62-94ac-dd9b801007e0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124448 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-sysconfig\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124474 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-etc-kubernetes\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124497 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-sysconfig\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124499 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-cni-netd\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124448 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/681bd531-3870-455f-bde7-39c25302c551-iptables-alerter-script\") pod \"iptables-alerter-xhtdn\" (UID: \"681bd531-3870-455f-bde7-39c25302c551\") " pod="openshift-network-operator/iptables-alerter-xhtdn" Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124524 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ffa4002-eab4-4d53-aad1-c82ba3810441-hosts-file\") pod \"node-resolver-pm6vt\" (UID: \"0ffa4002-eab4-4d53-aad1-c82ba3810441\") " pod="openshift-dns/node-resolver-pm6vt" Sep 11 13:57:41.124849 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124554 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-cni-netd\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124554 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-etc-kubernetes\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124583 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-registration-dir\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124602 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-run-netns\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124713 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-registration-dir\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124728 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-run-netns\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124765 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ffa4002-eab4-4d53-aad1-c82ba3810441-hosts-file\") pod \"node-resolver-pm6vt\" (UID: \"0ffa4002-eab4-4d53-aad1-c82ba3810441\") " pod="openshift-dns/node-resolver-pm6vt" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124784 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-systemd-units\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124803 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-run-netns\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124822 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-systemd-units\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124817 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-kubelet-config\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124846 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-run-netns\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124874 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275zb\" (UniqueName: \"kubernetes.io/projected/98e24790-f943-4aa7-b079-b8940fb02a82-kube-api-access-275zb\") pod \"network-metrics-daemon-hphpv\" (UID: \"98e24790-f943-4aa7-b079-b8940fb02a82\") " pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124893 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-kubernetes\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124907 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-lib-modules\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124923 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-cni-binary-copy\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124876 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-kubelet-config\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:41.125635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124946 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-hostroot\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.124970 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125000 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ffa4002-eab4-4d53-aad1-c82ba3810441-tmp-dir\") pod \"node-resolver-pm6vt\" (UID: \"0ffa4002-eab4-4d53-aad1-c82ba3810441\") " pod="openshift-dns/node-resolver-pm6vt" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125023 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125031 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-lib-modules\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125061 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs\") pod \"network-metrics-daemon-hphpv\" (UID: \"98e24790-f943-4aa7-b079-b8940fb02a82\") " pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125103 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125110 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-tuned\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125208 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125252 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-kubernetes\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.125349 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125387 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-hostroot\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125449 2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125469 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ffa4002-eab4-4d53-aad1-c82ba3810441-tmp-dir\") pod \"node-resolver-pm6vt\" (UID: \"0ffa4002-eab4-4d53-aad1-c82ba3810441\") " pod="openshift-dns/node-resolver-pm6vt" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125473 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-cni-binary-copy\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.125825 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs podName:98e24790-f943-4aa7-b079-b8940fb02a82 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:41.625432146 +0000 UTC m=+2.030413253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs") pod "network-metrics-daemon-hphpv" (UID: "98e24790-f943-4aa7-b079-b8940fb02a82") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125845 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7673f829-d14f-4abb-8c72-106d9dfd0a20-tmp\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125865 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-multus-cni-dir\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.126358 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125879 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-cni-bin\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125896 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkmlh\" (UniqueName: \"kubernetes.io/projected/49578359-6ee4-4d62-94ac-dd9b801007e0-kube-api-access-hkmlh\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125915 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl8kl\" (UniqueName: \"kubernetes.io/projected/7673f829-d14f-4abb-8c72-106d9dfd0a20-kube-api-access-sl8kl\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125952 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-sys-fs\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125963 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-cni-bin\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.125983 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ec98f86-dcb9-42cf-9add-e2045fbbe773-konnectivity-ca\") pod \"konnectivity-agent-zvnz6\" (UID: \"9ec98f86-dcb9-42cf-9add-e2045fbbe773\") " pod="kube-system/konnectivity-agent-zvnz6" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126010 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-run\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126040 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz4q6\" (UniqueName: \"kubernetes.io/projected/681bd531-3870-455f-bde7-39c25302c551-kube-api-access-kz4q6\") pod \"iptables-alerter-xhtdn\" (UID: \"681bd531-3870-455f-bde7-39c25302c551\") " pod="openshift-network-operator/iptables-alerter-xhtdn" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126063 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-sys-fs\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126067 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/94ee8113-90df-4db6-86d6-f042e909e9c7-ovn-node-metrics-cert\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126110 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/94ee8113-90df-4db6-86d6-f042e909e9c7-ovnkube-script-lib\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126136 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49578359-6ee4-4d62-94ac-dd9b801007e0-cni-binary-copy\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126160 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-kubelet\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126185 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-var-lib-openvswitch\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126215 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cngjm\" (UniqueName: \"kubernetes.io/projected/0ffa4002-eab4-4d53-aad1-c82ba3810441-kube-api-access-cngjm\") pod \"node-resolver-pm6vt\" (UID: \"0ffa4002-eab4-4d53-aad1-c82ba3810441\") " pod="openshift-dns/node-resolver-pm6vt" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126240 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49578359-6ee4-4d62-94ac-dd9b801007e0-system-cni-dir\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126262 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-cnibin\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.127263 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126285 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-var-lib-cni-bin\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126314 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-run-ovn\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126332 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-run\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126336 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-device-dir\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126357 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-var-lib-openvswitch\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126359 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4krc\" (UniqueName: \"kubernetes.io/projected/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-kube-api-access-f4krc\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126385 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-cnibin\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126395 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-multus-conf-dir\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126435 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-multus-daemon-config\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126459 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-run-systemd\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126480 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-node-log\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126533 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jxqf\" (UniqueName: \"kubernetes.io/projected/94ee8113-90df-4db6-86d6-f042e909e9c7-kube-api-access-9jxqf\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126569 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49578359-6ee4-4d62-94ac-dd9b801007e0-cnibin\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126597 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-run-ovn\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126590 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2c22dff-0209-4bda-9a7a-5626d2268974-host\") pod \"node-ca-mjmwj\" (UID: \"c2c22dff-0209-4bda-9a7a-5626d2268974\") " pod="openshift-image-registry/node-ca-mjmwj" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126661 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-device-dir\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126670 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2c22dff-0209-4bda-9a7a-5626d2268974-serviceca\") pod \"node-ca-mjmwj\" (UID: \"c2c22dff-0209-4bda-9a7a-5626d2268974\") " pod="openshift-image-registry/node-ca-mjmwj" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126676 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/49578359-6ee4-4d62-94ac-dd9b801007e0-system-cni-dir\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.128049 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126707 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-run-systemd\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126722 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-kubelet\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126740 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-node-log\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126745 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-run-multus-certs\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126708 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-run-multus-certs\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126766 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ec98f86-dcb9-42cf-9add-e2045fbbe773-konnectivity-ca\") pod \"konnectivity-agent-zvnz6\" (UID: \"9ec98f86-dcb9-42cf-9add-e2045fbbe773\") " pod="kube-system/konnectivity-agent-zvnz6" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126793 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126814 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-multus-conf-dir\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126827 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94ee8113-90df-4db6-86d6-f042e909e9c7-host-run-ovn-kubernetes\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126862 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/49578359-6ee4-4d62-94ac-dd9b801007e0-cnibin\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126874 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2c22dff-0209-4bda-9a7a-5626d2268974-host\") pod \"node-ca-mjmwj\" (UID: \"c2c22dff-0209-4bda-9a7a-5626d2268974\") " pod="openshift-image-registry/node-ca-mjmwj" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126040 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-multus-cni-dir\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126888 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49578359-6ee4-4d62-94ac-dd9b801007e0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126938 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ec98f86-dcb9-42cf-9add-e2045fbbe773-agent-certs\") pod \"konnectivity-agent-zvnz6\" (UID: \"9ec98f86-dcb9-42cf-9add-e2045fbbe773\") " pod="kube-system/konnectivity-agent-zvnz6" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.126961 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-sysctl-conf\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.127304 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-sysctl-conf\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.127356 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c2c22dff-0209-4bda-9a7a-5626d2268974-serviceca\") pod \"node-ca-mjmwj\" (UID: \"c2c22dff-0209-4bda-9a7a-5626d2268974\") " pod="openshift-image-registry/node-ca-mjmwj" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.127420 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-host-var-lib-cni-bin\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.128945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.127434 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/94ee8113-90df-4db6-86d6-f042e909e9c7-ovnkube-script-lib\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.129545 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.127690 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/49578359-6ee4-4d62-94ac-dd9b801007e0-cni-binary-copy\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.129545 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.127844 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-multus-daemon-config\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.129545 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.127850 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/49578359-6ee4-4d62-94ac-dd9b801007e0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.129545 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.129306 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7673f829-d14f-4abb-8c72-106d9dfd0a20-etc-tuned\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.129545 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.129346 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/94ee8113-90df-4db6-86d6-f042e909e9c7-ovn-node-metrics-cert\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.129545 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.129351 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7673f829-d14f-4abb-8c72-106d9dfd0a20-tmp\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.129846 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.129831 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ec98f86-dcb9-42cf-9add-e2045fbbe773-agent-certs\") pod \"konnectivity-agent-zvnz6\" (UID: \"9ec98f86-dcb9-42cf-9add-e2045fbbe773\") " pod="kube-system/konnectivity-agent-zvnz6" Sep 11 13:57:41.134417 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.134403 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:57:41.134471 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.134419 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:57:41.134471 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.134429 2566 projected.go:196] Error preparing data for projected volume kube-api-access-d8t5m for pod openshift-network-diagnostics/network-check-target-snsqs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:57:41.134529 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.134477 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m podName:ce87902b-b413-4c11-8704-f874a61e1f83 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:41.63446439 +0000 UTC m=+2.039445497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d8t5m" (UniqueName: "kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m") pod "network-check-target-snsqs" (UID: "ce87902b-b413-4c11-8704-f874a61e1f83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:57:41.137428 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.137409 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7kg2\" (UniqueName: \"kubernetes.io/projected/c2c22dff-0209-4bda-9a7a-5626d2268974-kube-api-access-t7kg2\") pod \"node-ca-mjmwj\" (UID: \"c2c22dff-0209-4bda-9a7a-5626d2268974\") " pod="openshift-image-registry/node-ca-mjmwj" Sep 11 13:57:41.137653 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.137633 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr9ks\" (UniqueName: \"kubernetes.io/projected/b3018b6c-24af-4ae4-a83b-c38a52fd7fc7-kube-api-access-dr9ks\") pod \"multus-rtct4\" (UID: \"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7\") " pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.140141 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.140121 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4krc\" (UniqueName: \"kubernetes.io/projected/5ca55b9d-ea58-41ec-9dd3-c25b413e98be-kube-api-access-f4krc\") pod \"aws-ebs-csi-driver-node-tpfnn\" (UID: \"5ca55b9d-ea58-41ec-9dd3-c25b413e98be\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.140511 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.140484 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz4q6\" (UniqueName: \"kubernetes.io/projected/681bd531-3870-455f-bde7-39c25302c551-kube-api-access-kz4q6\") pod \"iptables-alerter-xhtdn\" (UID: \"681bd531-3870-455f-bde7-39c25302c551\") " pod="openshift-network-operator/iptables-alerter-xhtdn" Sep 11 13:57:41.141337 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.141313 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cngjm\" (UniqueName: \"kubernetes.io/projected/0ffa4002-eab4-4d53-aad1-c82ba3810441-kube-api-access-cngjm\") pod \"node-resolver-pm6vt\" (UID: \"0ffa4002-eab4-4d53-aad1-c82ba3810441\") " pod="openshift-dns/node-resolver-pm6vt" Sep 11 13:57:41.141535 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.141515 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl8kl\" (UniqueName: \"kubernetes.io/projected/7673f829-d14f-4abb-8c72-106d9dfd0a20-kube-api-access-sl8kl\") pod \"tuned-pcwfb\" (UID: \"7673f829-d14f-4abb-8c72-106d9dfd0a20\") " pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.142153 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.142132 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkmlh\" (UniqueName: \"kubernetes.io/projected/49578359-6ee4-4d62-94ac-dd9b801007e0-kube-api-access-hkmlh\") pod \"multus-additional-cni-plugins-crfj6\" (UID: \"49578359-6ee4-4d62-94ac-dd9b801007e0\") " pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.143528 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.143492 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-275zb\" (UniqueName: \"kubernetes.io/projected/98e24790-f943-4aa7-b079-b8940fb02a82-kube-api-access-275zb\") pod \"network-metrics-daemon-hphpv\" (UID: \"98e24790-f943-4aa7-b079-b8940fb02a82\") " pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:41.144751 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.144734 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jxqf\" (UniqueName: \"kubernetes.io/projected/94ee8113-90df-4db6-86d6-f042e909e9c7-kube-api-access-9jxqf\") pod \"ovnkube-node-t6zm9\" (UID: \"94ee8113-90df-4db6-86d6-f042e909e9c7\") " pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.218794 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:41.218772 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504a2a5e8c9cf487f28426e21025e719.slice/crio-f627ac395449b0717152bb92945276340b5c8dc8a1cc158b51c87d834ea056bf WatchSource:0}: Error finding container f627ac395449b0717152bb92945276340b5c8dc8a1cc158b51c87d834ea056bf: Status 404 returned error can't find the container with id f627ac395449b0717152bb92945276340b5c8dc8a1cc158b51c87d834ea056bf Sep 11 13:57:41.313611 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.313533 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rtct4" Sep 11 13:57:41.321712 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:41.321691 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3018b6c_24af_4ae4_a83b_c38a52fd7fc7.slice/crio-d873b2882f889c9de550e29d6f3fec9f58f56234d132cdc7d917b16ba274fd93 WatchSource:0}: Error finding container d873b2882f889c9de550e29d6f3fec9f58f56234d132cdc7d917b16ba274fd93: Status 404 returned error can't find the container with id d873b2882f889c9de550e29d6f3fec9f58f56234d132cdc7d917b16ba274fd93 Sep 11 13:57:41.332725 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.332708 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" Sep 11 13:57:41.339149 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.339128 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:57:41.339368 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:41.339344 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7673f829_d14f_4abb_8c72_106d9dfd0a20.slice/crio-f00e255c789559af5a740b4fedc9bf232bda556c5aa269f0b167ea6acc002d4d WatchSource:0}: Error finding container f00e255c789559af5a740b4fedc9bf232bda556c5aa269f0b167ea6acc002d4d: Status 404 returned error can't find the container with id f00e255c789559af5a740b4fedc9bf232bda556c5aa269f0b167ea6acc002d4d Sep 11 13:57:41.342350 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.342330 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pm6vt" Sep 11 13:57:41.347841 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.347821 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-crfj6" Sep 11 13:57:41.348292 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:41.348276 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ee8113_90df_4db6_86d6_f042e909e9c7.slice/crio-ae963c120fd05ace990254e0b3555a9ffdf6f43dae32102207cfbacd7cacf0b6 WatchSource:0}: Error finding container ae963c120fd05ace990254e0b3555a9ffdf6f43dae32102207cfbacd7cacf0b6: Status 404 returned error can't find the container with id ae963c120fd05ace990254e0b3555a9ffdf6f43dae32102207cfbacd7cacf0b6 Sep 11 13:57:41.351702 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:41.351672 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ffa4002_eab4_4d53_aad1_c82ba3810441.slice/crio-3daed48f1e58918e03497426817a62e6b3123d446771b2fbe9b1627a1470ab82 WatchSource:0}: Error finding container 3daed48f1e58918e03497426817a62e6b3123d446771b2fbe9b1627a1470ab82: Status 404 returned error can't find the container with id 3daed48f1e58918e03497426817a62e6b3123d446771b2fbe9b1627a1470ab82 Sep 11 13:57:41.353153 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.353125 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" Sep 11 13:57:41.358206 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.358188 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zvnz6" Sep 11 13:57:41.363596 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.363566 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mjmwj" Sep 11 13:57:41.363856 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:41.363834 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca55b9d_ea58_41ec_9dd3_c25b413e98be.slice/crio-47b55fd8efbcfe615a0519e8c6ed4f56d879fc98390fbf4b1800f01147f395de WatchSource:0}: Error finding container 47b55fd8efbcfe615a0519e8c6ed4f56d879fc98390fbf4b1800f01147f395de: Status 404 returned error can't find the container with id 47b55fd8efbcfe615a0519e8c6ed4f56d879fc98390fbf4b1800f01147f395de Sep 11 13:57:41.366246 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:41.366220 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec98f86_dcb9_42cf_9add_e2045fbbe773.slice/crio-70f8fb917db9782d7db67b8e365a5f99b77bf4030f23243ab18678c5f716809a WatchSource:0}: Error finding container 70f8fb917db9782d7db67b8e365a5f99b77bf4030f23243ab18678c5f716809a: Status 404 returned error can't find the container with id 70f8fb917db9782d7db67b8e365a5f99b77bf4030f23243ab18678c5f716809a Sep 11 13:57:41.367264 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.367248 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xhtdn" Sep 11 13:57:41.372285 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:41.372266 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c22dff_0209_4bda_9a7a_5626d2268974.slice/crio-02d5e9216d96341cf3600674bb0707c53d7e1f294857548bb8ff9cfc5c8c285b WatchSource:0}: Error finding container 02d5e9216d96341cf3600674bb0707c53d7e1f294857548bb8ff9cfc5c8c285b: Status 404 returned error can't find the container with id 02d5e9216d96341cf3600674bb0707c53d7e1f294857548bb8ff9cfc5c8c285b Sep 11 13:57:41.376539 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:57:41.376519 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681bd531_3870_455f_bde7_39c25302c551.slice/crio-d8aca455098eadafb32c9f48af3204ad65c459db222699a9adba89de8731eceb WatchSource:0}: Error finding container d8aca455098eadafb32c9f48af3204ad65c459db222699a9adba89de8731eceb: Status 404 returned error can't find the container with id d8aca455098eadafb32c9f48af3204ad65c459db222699a9adba89de8731eceb Sep 11 13:57:41.629685 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.629614 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs\") pod \"network-metrics-daemon-hphpv\" (UID: \"98e24790-f943-4aa7-b079-b8940fb02a82\") " pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:41.629685 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.629676 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:41.629888 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.629781 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:57:41.629888 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.629795 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:57:41.629888 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.629848 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret podName:3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:42.629830209 +0000 UTC m=+3.034811328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret") pod "global-pull-secret-syncer-4wl44" (UID: "3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:57:41.629888 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.629881 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs podName:98e24790-f943-4aa7-b079-b8940fb02a82 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:42.62985838 +0000 UTC m=+3.034839491 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs") pod "network-metrics-daemon-hphpv" (UID: "98e24790-f943-4aa7-b079-b8940fb02a82") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:57:41.730725 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.730687 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8t5m\" (UniqueName: \"kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m\") pod \"network-check-target-snsqs\" (UID: \"ce87902b-b413-4c11-8704-f874a61e1f83\") " pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:41.730894 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.730873 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:57:41.730937 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.730900 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:57:41.730937 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.730911 2566 projected.go:196] Error preparing data for projected volume kube-api-access-d8t5m for pod openshift-network-diagnostics/network-check-target-snsqs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:57:41.731014 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:41.730976 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m podName:ce87902b-b413-4c11-8704-f874a61e1f83 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:42.730959721 +0000 UTC m=+3.135940830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8t5m" (UniqueName: "kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m") pod "network-check-target-snsqs" (UID: "ce87902b-b413-4c11-8704-f874a61e1f83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:57:41.937449 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:41.937369 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:57:42.081725 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.081673 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 13:52:41 +0000 UTC" deadline="2028-06-19 19:38:53.602108006 +0000 UTC" Sep 11 13:57:42.081725 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.081715 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15533h41m11.520398057s" Sep 11 13:57:42.121816 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.121709 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mjmwj" event={"ID":"c2c22dff-0209-4bda-9a7a-5626d2268974","Type":"ContainerStarted","Data":"02d5e9216d96341cf3600674bb0707c53d7e1f294857548bb8ff9cfc5c8c285b"} Sep 11 13:57:42.126750 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.126705 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" event={"ID":"5ca55b9d-ea58-41ec-9dd3-c25b413e98be","Type":"ContainerStarted","Data":"47b55fd8efbcfe615a0519e8c6ed4f56d879fc98390fbf4b1800f01147f395de"} Sep 11 13:57:42.138219 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.138186 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crfj6" event={"ID":"49578359-6ee4-4d62-94ac-dd9b801007e0","Type":"ContainerStarted","Data":"420ba6194d3f05780c3deb2936832cff35b1dee338b6277b41d83d1adaa7cd0d"} Sep 11 13:57:42.152106 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.152076 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pm6vt" event={"ID":"0ffa4002-eab4-4d53-aad1-c82ba3810441","Type":"ContainerStarted","Data":"3daed48f1e58918e03497426817a62e6b3123d446771b2fbe9b1627a1470ab82"} Sep 11 13:57:42.171803 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.171766 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" event={"ID":"7673f829-d14f-4abb-8c72-106d9dfd0a20","Type":"ContainerStarted","Data":"f00e255c789559af5a740b4fedc9bf232bda556c5aa269f0b167ea6acc002d4d"} Sep 11 13:57:42.177757 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.177727 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xhtdn" event={"ID":"681bd531-3870-455f-bde7-39c25302c551","Type":"ContainerStarted","Data":"d8aca455098eadafb32c9f48af3204ad65c459db222699a9adba89de8731eceb"} Sep 11 13:57:42.186850 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.186819 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zvnz6" event={"ID":"9ec98f86-dcb9-42cf-9add-e2045fbbe773","Type":"ContainerStarted","Data":"70f8fb917db9782d7db67b8e365a5f99b77bf4030f23243ab18678c5f716809a"} Sep 11 13:57:42.207560 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.207479 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" event={"ID":"94ee8113-90df-4db6-86d6-f042e909e9c7","Type":"ContainerStarted","Data":"ae963c120fd05ace990254e0b3555a9ffdf6f43dae32102207cfbacd7cacf0b6"} Sep 11 13:57:42.221036 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.220999 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rtct4" event={"ID":"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7","Type":"ContainerStarted","Data":"d873b2882f889c9de550e29d6f3fec9f58f56234d132cdc7d917b16ba274fd93"} Sep 11 13:57:42.222737 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.222668 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-92.ec2.internal" event={"ID":"504a2a5e8c9cf487f28426e21025e719","Type":"ContainerStarted","Data":"f627ac395449b0717152bb92945276340b5c8dc8a1cc158b51c87d834ea056bf"} Sep 11 13:57:42.378636 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.378402 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:57:42.578241 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.577947 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:57:42.640232 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.640193 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:42.640439 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.640255 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs\") pod \"network-metrics-daemon-hphpv\" (UID: \"98e24790-f943-4aa7-b079-b8940fb02a82\") " pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:42.640439 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:42.640422 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:57:42.640562 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:42.640532 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs podName:98e24790-f943-4aa7-b079-b8940fb02a82 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:44.640508983 +0000 UTC m=+5.045490100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs") pod "network-metrics-daemon-hphpv" (UID: "98e24790-f943-4aa7-b079-b8940fb02a82") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:57:42.641130 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:42.641110 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:57:42.641227 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:42.641186 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret podName:3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:44.641154408 +0000 UTC m=+5.046135518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret") pod "global-pull-secret-syncer-4wl44" (UID: "3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:57:42.741068 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:42.741031 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8t5m\" (UniqueName: \"kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m\") pod \"network-check-target-snsqs\" (UID: \"ce87902b-b413-4c11-8704-f874a61e1f83\") " pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:42.741280 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:42.741198 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:57:42.741280 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:42.741218 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:57:42.741280 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:42.741230 2566 projected.go:196] Error preparing data for projected volume kube-api-access-d8t5m for pod openshift-network-diagnostics/network-check-target-snsqs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:57:42.741451 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:42.741286 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m podName:ce87902b-b413-4c11-8704-f874a61e1f83 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:44.741267989 +0000 UTC m=+5.146249097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8t5m" (UniqueName: "kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m") pod "network-check-target-snsqs" (UID: "ce87902b-b413-4c11-8704-f874a61e1f83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:57:43.082452 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:43.082176 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 13:52:41 +0000 UTC" deadline="2028-06-13 10:32:08.90556149 +0000 UTC" Sep 11 13:57:43.082452 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:43.082453 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15380h34m25.823113652s" Sep 11 13:57:43.091273 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:43.091235 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:43.091448 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:43.091372 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:57:43.091805 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:43.091785 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:43.091922 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:43.091885 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:57:43.091987 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:43.091948 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:43.092049 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:43.092031 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:57:44.658690 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:44.658656 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:44.659054 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:44.658712 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs\") pod \"network-metrics-daemon-hphpv\" (UID: \"98e24790-f943-4aa7-b079-b8940fb02a82\") " pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:44.659054 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:44.658815 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:57:44.659054 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:44.658868 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:57:44.659054 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:44.658874 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret podName:3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:48.658859528 +0000 UTC m=+9.063840638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret") pod "global-pull-secret-syncer-4wl44" (UID: "3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:57:44.659054 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:44.658939 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs podName:98e24790-f943-4aa7-b079-b8940fb02a82 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:48.658922019 +0000 UTC m=+9.063903125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs") pod "network-metrics-daemon-hphpv" (UID: "98e24790-f943-4aa7-b079-b8940fb02a82") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:57:44.759471 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:44.759430 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8t5m\" (UniqueName: \"kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m\") pod \"network-check-target-snsqs\" (UID: \"ce87902b-b413-4c11-8704-f874a61e1f83\") " pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:44.759657 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:44.759604 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:57:44.759657 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:44.759623 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:57:44.759657 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:44.759635 2566 projected.go:196] Error preparing data for projected volume kube-api-access-d8t5m for pod openshift-network-diagnostics/network-check-target-snsqs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:57:44.759823 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:44.759694 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m podName:ce87902b-b413-4c11-8704-f874a61e1f83 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:48.759676183 +0000 UTC m=+9.164657304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8t5m" (UniqueName: "kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m") pod "network-check-target-snsqs" (UID: "ce87902b-b413-4c11-8704-f874a61e1f83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:57:45.092617 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:45.091852 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:45.092617 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:45.091977 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:57:45.092617 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:45.092338 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:45.092617 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:45.092407 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:57:45.092617 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:45.092458 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:45.092617 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:45.092538 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:57:47.092315 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:47.091802 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:47.092315 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:47.091944 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:57:47.092315 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:47.092029 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:47.092315 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:47.092180 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:57:47.092315 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:47.092224 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:47.092315 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:47.092295 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:57:48.697430 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:48.697350 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:48.697430 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:48.697406 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs\") pod \"network-metrics-daemon-hphpv\" (UID: \"98e24790-f943-4aa7-b079-b8940fb02a82\") " pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:48.697896 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:48.697570 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:57:48.697896 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:48.697636 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs podName:98e24790-f943-4aa7-b079-b8940fb02a82 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:56.697615658 +0000 UTC m=+17.102596764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs") pod "network-metrics-daemon-hphpv" (UID: "98e24790-f943-4aa7-b079-b8940fb02a82") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:57:48.698068 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:48.698048 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:57:48.698160 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:48.698128 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret podName:3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:56.698112856 +0000 UTC m=+17.103093963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret") pod "global-pull-secret-syncer-4wl44" (UID: "3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:57:48.798455 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:48.798416 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8t5m\" (UniqueName: \"kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m\") pod \"network-check-target-snsqs\" (UID: \"ce87902b-b413-4c11-8704-f874a61e1f83\") " pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:48.798626 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:48.798578 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:57:48.798626 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:48.798609 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:57:48.798626 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:48.798622 2566 projected.go:196] Error preparing data for projected volume kube-api-access-d8t5m for pod openshift-network-diagnostics/network-check-target-snsqs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:57:48.798797 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:48.798685 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m podName:ce87902b-b413-4c11-8704-f874a61e1f83 nodeName:}" failed. No retries permitted until 2026-09-11 13:57:56.798665839 +0000 UTC m=+17.203646968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8t5m" (UniqueName: "kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m") pod "network-check-target-snsqs" (UID: "ce87902b-b413-4c11-8704-f874a61e1f83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:57:49.091430 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:49.091348 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:49.091559 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:49.091487 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:57:49.091848 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:49.091821 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:49.091948 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:49.091833 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:49.091998 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:49.091959 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:57:49.092060 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:49.092021 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:57:51.091493 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:51.091458 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:51.091870 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:51.091499 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:51.091870 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:51.091573 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:57:51.091870 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:51.091601 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:51.091870 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:51.091719 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:57:51.091870 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:51.091803 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:57:53.091750 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:53.091713 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:53.091750 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:53.091754 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:53.092341 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:53.091760 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:53.092341 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:53.091833 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:57:53.092341 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:53.091913 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:57:53.092341 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:53.092015 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:57:55.091864 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:55.091823 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:55.092303 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:55.091925 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:55.092303 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:55.091938 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:57:55.092303 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:55.092009 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:57:55.092303 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:55.092021 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:55.092303 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:55.092150 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:57:56.762274 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:56.762242 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:56.762274 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:56.762281 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs\") pod \"network-metrics-daemon-hphpv\" (UID: \"98e24790-f943-4aa7-b079-b8940fb02a82\") " pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:56.762684 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:56.762374 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:57:56.762684 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:56.762382 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:57:56.762684 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:56.762423 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs podName:98e24790-f943-4aa7-b079-b8940fb02a82 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:12.762409759 +0000 UTC m=+33.167390865 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs") pod "network-metrics-daemon-hphpv" (UID: "98e24790-f943-4aa7-b079-b8940fb02a82") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:57:56.762684 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:56.762436 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret podName:3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:12.762430086 +0000 UTC m=+33.167411192 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret") pod "global-pull-secret-syncer-4wl44" (UID: "3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:57:56.863365 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:56.863329 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8t5m\" (UniqueName: \"kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m\") pod \"network-check-target-snsqs\" (UID: \"ce87902b-b413-4c11-8704-f874a61e1f83\") " pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:56.863535 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:56.863514 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:57:56.863586 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:56.863546 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:57:56.863586 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:56.863560 2566 projected.go:196] Error preparing data for projected volume kube-api-access-d8t5m for pod openshift-network-diagnostics/network-check-target-snsqs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:57:56.863661 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:56.863627 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m podName:ce87902b-b413-4c11-8704-f874a61e1f83 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:12.863610007 +0000 UTC m=+33.268591117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8t5m" (UniqueName: "kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m") pod "network-check-target-snsqs" (UID: "ce87902b-b413-4c11-8704-f874a61e1f83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:57:57.091280 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:57.091180 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:57.091280 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:57.091204 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:57.091280 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:57.091193 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:57.091495 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:57.091315 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:57:57.091495 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:57.091384 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:57:57.091495 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:57.091473 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:57:59.092118 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:59.091687 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:57:59.092118 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:59.091718 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:57:59.092118 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:57:59.091724 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:57:59.092118 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:59.091797 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:57:59.092118 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:59.091885 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:57:59.092118 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:57:59.091958 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:58:00.275398 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:00.274754 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" event={"ID":"7673f829-d14f-4abb-8c72-106d9dfd0a20","Type":"ContainerStarted","Data":"fbff201ac3ccef9c75e0763f34fa67266c9c827fb5b4ab74d6533203f09f795a"} Sep 11 13:58:00.280376 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:00.280208 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" event={"ID":"94ee8113-90df-4db6-86d6-f042e909e9c7","Type":"ContainerStarted","Data":"ca22f2b17c7fbe022cf3cbb190bf0c44e535b0a37acf2fdc30ef03b6719f8edb"} Sep 11 13:58:00.282688 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:00.282653 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rtct4" event={"ID":"b3018b6c-24af-4ae4-a83b-c38a52fd7fc7","Type":"ContainerStarted","Data":"d30c15d2f2f947ccbad6cc01e05956a669ed2ee960088ea8586056f347bf9367"} Sep 11 13:58:00.291316 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:00.286783 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-92.ec2.internal" event={"ID":"504a2a5e8c9cf487f28426e21025e719","Type":"ContainerStarted","Data":"26ca0b77b42f85cb78bedbac597282979a381c9e517a9857c802733cacd8ae32"} Sep 11 13:58:00.301115 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:00.300918 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pcwfb" podStartSLOduration=1.578661051 podStartE2EDuration="20.300895236s" totalImagesPullingTime="18.722234185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:00.300180291 +0000 UTC m=+20.705161421" watchObservedRunningTime="2026-09-11 13:58:00.300895236 +0000 UTC m=+20.705876366" Sep 11 13:58:00.336912 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:00.336745 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rtct4" podStartSLOduration=1.5951990569999999 podStartE2EDuration="20.336728155s" totalImagesPullingTime="18.741529098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:00.336305849 +0000 UTC m=+20.741286978" watchObservedRunningTime="2026-09-11 13:58:00.336728155 +0000 UTC m=+20.741709266" Sep 11 13:58:00.337395 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:00.337272 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-92.ec2.internal" podStartSLOduration=19.337257713 podStartE2EDuration="19.337257713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:00.318181886 +0000 UTC m=+20.723163017" watchObservedRunningTime="2026-09-11 13:58:00.337257713 +0000 UTC m=+20.742238842" Sep 11 13:58:01.092047 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.091844 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:58:01.092206 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:01.092161 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:58:01.092206 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.091926 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:58:01.092321 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:01.092285 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:58:01.092625 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.091956 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:58:01.092740 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:01.092703 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:58:01.290265 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.290229 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zvnz6" event={"ID":"9ec98f86-dcb9-42cf-9add-e2045fbbe773","Type":"ContainerStarted","Data":"587ed98ea010f5ae0b324c32227a4aba176c5f3ee800e8133860887322f0de0e"} Sep 11 13:58:01.293393 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.293365 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" event={"ID":"94ee8113-90df-4db6-86d6-f042e909e9c7","Type":"ContainerStarted","Data":"219bff74fc9c43174e062fbe61f73926832ba449ef8c5b708a0f1c44cc3f39dd"} Sep 11 13:58:01.293513 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.293440 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" event={"ID":"94ee8113-90df-4db6-86d6-f042e909e9c7","Type":"ContainerStarted","Data":"55a0df1e7557907ef886661765c9479a860ac44622b715f7ca7550f7484e9eaa"} Sep 11 13:58:01.293513 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.293490 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" event={"ID":"94ee8113-90df-4db6-86d6-f042e909e9c7","Type":"ContainerStarted","Data":"261fed67e0bd9d656240be3ba6a82a30daa1029c7ed2e0fd7f73b81f6a171d04"} Sep 11 13:58:01.293626 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.293539 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" event={"ID":"94ee8113-90df-4db6-86d6-f042e909e9c7","Type":"ContainerStarted","Data":"e33d75dcffe1c2e90ab265a676db782b6c7f8f9e6fb239480383cb02e2cb1775"} Sep 11 13:58:01.293626 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.293586 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" event={"ID":"94ee8113-90df-4db6-86d6-f042e909e9c7","Type":"ContainerStarted","Data":"0bffe4f6d72f39b0cb269d21542e44922357c5fc690fcbb960b352276b3b4852"} Sep 11 13:58:01.294918 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.294893 2566 generic.go:356] "Generic (PLEG): container finished" podID="4f35c18b2576be95c1be41030bb842c9" containerID="ef81be9f079bd1e81bb32b72723c5f04054b88a20000aa00db9d67aa90c67e58" exitCode=0 Sep 11 13:58:01.295010 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.294958 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal" event={"ID":"4f35c18b2576be95c1be41030bb842c9","Type":"ContainerDied","Data":"ef81be9f079bd1e81bb32b72723c5f04054b88a20000aa00db9d67aa90c67e58"} Sep 11 13:58:01.296551 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.296527 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mjmwj" event={"ID":"c2c22dff-0209-4bda-9a7a-5626d2268974","Type":"ContainerStarted","Data":"4b8169379d34bd4aef07ef66e676e7f9c99d731857ee01b8c538cf7cf911aa66"} Sep 11 13:58:01.297932 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.297910 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" event={"ID":"5ca55b9d-ea58-41ec-9dd3-c25b413e98be","Type":"ContainerStarted","Data":"2818fc60837a66b56351df92131e52f3ab42bbfd24d506e3e997dece19dd4b14"} Sep 11 13:58:01.299786 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.299758 2566 generic.go:356] "Generic (PLEG): container finished" podID="49578359-6ee4-4d62-94ac-dd9b801007e0" containerID="8f81e984eea7a2cee7a7431989b8ee4b4066b7c21e0d097c316cf773127402d2" exitCode=0 Sep 11 13:58:01.299888 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.299805 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crfj6" event={"ID":"49578359-6ee4-4d62-94ac-dd9b801007e0","Type":"ContainerDied","Data":"8f81e984eea7a2cee7a7431989b8ee4b4066b7c21e0d097c316cf773127402d2"} Sep 11 13:58:01.301567 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.301483 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pm6vt" event={"ID":"0ffa4002-eab4-4d53-aad1-c82ba3810441","Type":"ContainerStarted","Data":"0209a016cbb8a259d5d26ebd0109192c1748af3c15df824b9358c9a28f9f67a2"} Sep 11 13:58:01.304830 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.304794 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zvnz6" podStartSLOduration=2.611369771 podStartE2EDuration="21.304782137s" totalImagesPullingTime="18.693412366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:01.304397412 +0000 UTC m=+21.709378539" watchObservedRunningTime="2026-09-11 13:58:01.304782137 +0000 UTC m=+21.709763264" Sep 11 13:58:01.352270 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.352173 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mjmwj" podStartSLOduration=2.664647923 podStartE2EDuration="21.35215622s" totalImagesPullingTime="18.687508297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:01.337316122 +0000 UTC m=+21.742297250" watchObservedRunningTime="2026-09-11 13:58:01.35215622 +0000 UTC m=+21.757137347" Sep 11 13:58:01.864846 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:01.864790 2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 13:58:02.093031 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:02.092946 2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T13:58:01.86481219Z","UUID":"dd681370-205f-40c3-8331-265886bf85ad","Handler":null,"Name":"","Endpoint":""} Sep 11 13:58:02.095791 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:02.095757 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 13:58:02.095791 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:02.095794 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 13:58:02.305844 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:02.305806 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xhtdn" event={"ID":"681bd531-3870-455f-bde7-39c25302c551","Type":"ContainerStarted","Data":"6fef188e556520894ba3d71f26263110882ed399b95a6d19220d143d82d6cb19"} Sep 11 13:58:02.308862 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:02.308831 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal" event={"ID":"4f35c18b2576be95c1be41030bb842c9","Type":"ContainerStarted","Data":"7c11367521af79ca51408a56fc19c8f24468a716cc25f3e0ce4acaa25e4d3f89"} Sep 11 13:58:02.311016 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:02.310993 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" event={"ID":"5ca55b9d-ea58-41ec-9dd3-c25b413e98be","Type":"ContainerStarted","Data":"5c82a14bc427615322ab8c3a1fe080ea572bc50219cce2d8eae3a60e208ba43e"} Sep 11 13:58:02.322593 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:02.322544 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xhtdn" podStartSLOduration=3.637646562 podStartE2EDuration="22.322530944s" totalImagesPullingTime="18.684884382s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:02.322349271 +0000 UTC m=+22.727330413" watchObservedRunningTime="2026-09-11 13:58:02.322530944 +0000 UTC m=+22.727512072" Sep 11 13:58:02.323168 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:02.323134 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pm6vt" podStartSLOduration=3.61494251 podStartE2EDuration="22.323125143s" totalImagesPullingTime="18.708182633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:01.372690819 +0000 UTC m=+21.777671946" watchObservedRunningTime="2026-09-11 13:58:02.323125143 +0000 UTC m=+22.728106320" Sep 11 13:58:02.342364 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:02.342322 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-92.ec2.internal" podStartSLOduration=21.342311582 podStartE2EDuration="21.342311582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:02.342063556 +0000 UTC m=+22.747044685" watchObservedRunningTime="2026-09-11 13:58:02.342311582 +0000 UTC m=+22.747292709" Sep 11 13:58:02.633811 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:02.633730 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zvnz6" Sep 11 13:58:02.634446 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:02.634413 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zvnz6" Sep 11 13:58:03.091765 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:03.091657 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:58:03.091765 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:03.091726 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:58:03.092005 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:03.091686 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:58:03.092005 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:03.091826 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:58:03.092005 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:03.091884 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:58:03.092005 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:03.091972 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:58:03.318457 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:03.318412 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" event={"ID":"94ee8113-90df-4db6-86d6-f042e909e9c7","Type":"ContainerStarted","Data":"f352f645660e8ccac32e0bbffaf4c1f8b248fe89336bd06e3640ca3348b052c3"} Sep 11 13:58:04.323893 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:04.323686 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" event={"ID":"5ca55b9d-ea58-41ec-9dd3-c25b413e98be","Type":"ContainerStarted","Data":"e78f92cc22382ba21ac0dbd910de2b140c304d4700658b7b6f68d4956364cacf"} Sep 11 13:58:04.324354 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:04.323699 2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 13:58:04.343344 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:04.343292 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpfnn" podStartSLOduration=3.037004629 podStartE2EDuration="24.343275934s" totalImagesPullingTime="21.306271305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:04.343066673 +0000 UTC m=+24.748047802" watchObservedRunningTime="2026-09-11 13:58:04.343275934 +0000 UTC m=+24.748257063" Sep 11 13:58:05.092922 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:05.091934 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:58:05.092922 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:05.091960 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:58:05.092922 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:05.091963 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:58:05.092922 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:05.092433 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:58:05.092922 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:05.092588 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:58:05.093311 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:05.092705 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:58:05.333294 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:05.333267 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" event={"ID":"94ee8113-90df-4db6-86d6-f042e909e9c7","Type":"ContainerStarted","Data":"65d09dddcfa47a01cd5eabd7ca944a6cbaa40124d43b22e072c6b7b3bd8b4951"} Sep 11 13:58:05.362712 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:05.362216 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" podStartSLOduration=6.570617304 podStartE2EDuration="25.362196688s" totalImagesPullingTime="18.791579384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:05.361947061 +0000 UTC m=+25.766928226" watchObservedRunningTime="2026-09-11 13:58:05.362196688 +0000 UTC m=+25.767177815" Sep 11 13:58:06.337187 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:06.337153 2566 generic.go:356] "Generic (PLEG): container finished" podID="49578359-6ee4-4d62-94ac-dd9b801007e0" containerID="66b3163204b893ff7dd04cff4190d5df2a768ff37100551abc0d0393721c019b" exitCode=0 Sep 11 13:58:06.337566 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:06.337192 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crfj6" event={"ID":"49578359-6ee4-4d62-94ac-dd9b801007e0","Type":"ContainerDied","Data":"66b3163204b893ff7dd04cff4190d5df2a768ff37100551abc0d0393721c019b"} Sep 11 13:58:06.338515 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:06.337788 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:58:06.338515 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:06.337844 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:58:06.338515 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:06.337880 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:58:06.353685 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:06.353662 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:58:06.354220 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:06.354205 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:58:06.944411 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:06.944382 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zvnz6" Sep 11 13:58:06.944543 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:06.944531 2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 13:58:06.944911 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:06.944888 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zvnz6" Sep 11 13:58:07.091166 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:07.091133 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:58:07.091166 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:07.091150 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:58:07.091166 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:07.091160 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:58:07.091389 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:07.091258 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:58:07.091389 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:07.091296 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:58:07.091389 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:07.091340 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:58:07.162075 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:07.162039 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hphpv"] Sep 11 13:58:07.165563 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:07.165536 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4wl44"] Sep 11 13:58:07.166310 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:07.166283 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-snsqs"] Sep 11 13:58:07.340157 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:07.340129 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:58:07.340157 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:07.340129 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:58:07.340790 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:07.340330 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:58:07.340790 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:07.340450 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:58:07.341133 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:07.341109 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:58:07.341252 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:07.341226 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:58:08.344533 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:08.344497 2566 generic.go:356] "Generic (PLEG): container finished" podID="49578359-6ee4-4d62-94ac-dd9b801007e0" containerID="7e2ef5046299a3c5028f8efc7307e35fd36ec09acc77c73e8fc27e525e2dc12e" exitCode=0 Sep 11 13:58:08.344986 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:08.344576 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crfj6" event={"ID":"49578359-6ee4-4d62-94ac-dd9b801007e0","Type":"ContainerDied","Data":"7e2ef5046299a3c5028f8efc7307e35fd36ec09acc77c73e8fc27e525e2dc12e"} Sep 11 13:58:09.090970 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:09.090944 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:58:09.091050 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:09.091023 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:58:09.091050 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:09.091043 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:58:09.091166 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:09.091149 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:58:09.091293 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:09.091275 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:58:09.091374 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:09.091354 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:58:09.353184 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:09.352966 2566 generic.go:356] "Generic (PLEG): container finished" podID="49578359-6ee4-4d62-94ac-dd9b801007e0" containerID="cc4ec2148db9ac207eb0b0f0e6adc19a8c3eb77314679d360e0580ab79a7256e" exitCode=0 Sep 11 13:58:09.353184 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:09.353049 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crfj6" event={"ID":"49578359-6ee4-4d62-94ac-dd9b801007e0","Type":"ContainerDied","Data":"cc4ec2148db9ac207eb0b0f0e6adc19a8c3eb77314679d360e0580ab79a7256e"} Sep 11 13:58:11.091326 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:11.091295 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:58:11.091727 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:11.091397 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-snsqs" podUID="ce87902b-b413-4c11-8704-f874a61e1f83" Sep 11 13:58:11.091727 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:11.091421 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:58:11.091727 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:11.091305 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:58:11.091727 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:11.091516 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4wl44" podUID="3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9" Sep 11 13:58:11.091727 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:11.091595 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hphpv" podUID="98e24790-f943-4aa7-b079-b8940fb02a82" Sep 11 13:58:12.770830 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:12.770786 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:58:12.771497 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:12.770845 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs\") pod \"network-metrics-daemon-hphpv\" (UID: \"98e24790-f943-4aa7-b079-b8940fb02a82\") " pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:58:12.771497 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:12.770954 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:58:12.771497 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:12.770955 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:58:12.771497 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:12.771014 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs podName:98e24790-f943-4aa7-b079-b8940fb02a82 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:44.770998001 +0000 UTC m=+65.175979128 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs") pod "network-metrics-daemon-hphpv" (UID: "98e24790-f943-4aa7-b079-b8940fb02a82") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:58:12.771497 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:12.771035 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret podName:3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:44.771024293 +0000 UTC m=+65.176005401 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret") pod "global-pull-secret-syncer-4wl44" (UID: "3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:58:12.871208 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:12.871111 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8t5m\" (UniqueName: \"kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m\") pod \"network-check-target-snsqs\" (UID: \"ce87902b-b413-4c11-8704-f874a61e1f83\") " pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:58:12.871434 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:12.871313 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:58:12.871434 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:12.871332 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:58:12.871434 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:12.871346 2566 projected.go:196] Error preparing data for projected volume kube-api-access-d8t5m for pod openshift-network-diagnostics/network-check-target-snsqs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:58:12.871434 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:12.871400 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m podName:ce87902b-b413-4c11-8704-f874a61e1f83 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:44.871382443 +0000 UTC m=+65.276363561 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8t5m" (UniqueName: "kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m") pod "network-check-target-snsqs" (UID: "ce87902b-b413-4c11-8704-f874a61e1f83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:58:12.923579 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:12.923555 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-92.ec2.internal" event="NodeReady" Sep 11 13:58:12.926320 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:12.926275 2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 13:58:12.976902 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:12.976870 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zpsfz"] Sep 11 13:58:13.005867 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.005834 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gwb7j"] Sep 11 13:58:13.006025 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.005963 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:13.008415 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.008392 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tbm52\"" Sep 11 13:58:13.008415 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.008392 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 13:58:13.008614 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.008547 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 13:58:13.021824 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.021799 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zpsfz"] Sep 11 13:58:13.021965 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.021848 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gwb7j"] Sep 11 13:58:13.021965 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.021939 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gwb7j" Sep 11 13:58:13.024636 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.024618 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 13:58:13.024787 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.024642 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 13:58:13.025028 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.025009 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rh6t5\"" Sep 11 13:58:13.025147 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.025076 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 13:58:13.072664 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.072626 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-config-volume\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:13.072850 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.072700 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:13.072850 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.072734 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-tmp-dir\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:13.072850 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.072770 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tszhb\" (UniqueName: \"kubernetes.io/projected/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-kube-api-access-tszhb\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:13.091214 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.091182 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:58:13.091352 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.091227 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:58:13.091352 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.091228 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:58:13.094053 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.094032 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 13:58:13.094195 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.094052 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qh858\"" Sep 11 13:58:13.094195 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.094036 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 13:58:13.094335 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.094321 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 13:58:13.094397 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.094334 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fgrdt\"" Sep 11 13:58:13.094449 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.094418 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 13:58:13.173712 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.173638 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-config-volume\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:13.173712 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.173700 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08e2fadb-d68b-469d-a7f7-6356a9907e6c-cert\") pod \"ingress-canary-gwb7j\" (UID: \"08e2fadb-d68b-469d-a7f7-6356a9907e6c\") " pod="openshift-ingress-canary/ingress-canary-gwb7j" Sep 11 13:58:13.173983 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.173726 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g79k5\" (UniqueName: \"kubernetes.io/projected/08e2fadb-d68b-469d-a7f7-6356a9907e6c-kube-api-access-g79k5\") pod \"ingress-canary-gwb7j\" (UID: \"08e2fadb-d68b-469d-a7f7-6356a9907e6c\") " pod="openshift-ingress-canary/ingress-canary-gwb7j" Sep 11 13:58:13.173983 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.173798 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:13.173983 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.173832 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-tmp-dir\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:13.173983 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:13.173899 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 13:58:13.173983 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.173951 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tszhb\" (UniqueName: \"kubernetes.io/projected/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-kube-api-access-tszhb\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:13.173983 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:13.173976 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls podName:d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:13.673958986 +0000 UTC m=+34.078940092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls") pod "dns-default-zpsfz" (UID: "d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8") : secret "dns-default-metrics-tls" not found Sep 11 13:58:13.174268 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.174191 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-tmp-dir\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:13.174304 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.174284 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-config-volume\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:13.186593 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.186574 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tszhb\" (UniqueName: \"kubernetes.io/projected/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-kube-api-access-tszhb\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:13.274850 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.274813 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08e2fadb-d68b-469d-a7f7-6356a9907e6c-cert\") pod \"ingress-canary-gwb7j\" (UID: \"08e2fadb-d68b-469d-a7f7-6356a9907e6c\") " pod="openshift-ingress-canary/ingress-canary-gwb7j" Sep 11 13:58:13.274850 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.274852 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g79k5\" (UniqueName: \"kubernetes.io/projected/08e2fadb-d68b-469d-a7f7-6356a9907e6c-kube-api-access-g79k5\") pod \"ingress-canary-gwb7j\" (UID: \"08e2fadb-d68b-469d-a7f7-6356a9907e6c\") " pod="openshift-ingress-canary/ingress-canary-gwb7j" Sep 11 13:58:13.277728 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.277701 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08e2fadb-d68b-469d-a7f7-6356a9907e6c-cert\") pod \"ingress-canary-gwb7j\" (UID: \"08e2fadb-d68b-469d-a7f7-6356a9907e6c\") " pod="openshift-ingress-canary/ingress-canary-gwb7j" Sep 11 13:58:13.285738 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.285707 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g79k5\" (UniqueName: \"kubernetes.io/projected/08e2fadb-d68b-469d-a7f7-6356a9907e6c-kube-api-access-g79k5\") pod \"ingress-canary-gwb7j\" (UID: \"08e2fadb-d68b-469d-a7f7-6356a9907e6c\") " pod="openshift-ingress-canary/ingress-canary-gwb7j" Sep 11 13:58:13.331728 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.331692 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gwb7j" Sep 11 13:58:13.677807 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.677773 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:13.677948 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:13.677890 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 13:58:13.677948 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:13.677942 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls podName:d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:14.677928321 +0000 UTC m=+35.082909428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls") pod "dns-default-zpsfz" (UID: "d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8") : secret "dns-default-metrics-tls" not found Sep 11 13:58:13.919592 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:13.919397 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gwb7j"] Sep 11 13:58:13.923031 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:58:13.923005 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e2fadb_d68b_469d_a7f7_6356a9907e6c.slice/crio-36ebd4c1430a5d5646a0425004677be1ba0212a1c00082be0dad7928870aa0ad WatchSource:0}: Error finding container 36ebd4c1430a5d5646a0425004677be1ba0212a1c00082be0dad7928870aa0ad: Status 404 returned error can't find the container with id 36ebd4c1430a5d5646a0425004677be1ba0212a1c00082be0dad7928870aa0ad Sep 11 13:58:14.368027 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:14.367994 2566 generic.go:356] "Generic (PLEG): container finished" podID="49578359-6ee4-4d62-94ac-dd9b801007e0" containerID="5d006c5bd28eb6ccb7dc47c227b6b2f972e651b430df97d6d54dc8803635d9ab" exitCode=0 Sep 11 13:58:14.368223 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:14.368099 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crfj6" event={"ID":"49578359-6ee4-4d62-94ac-dd9b801007e0","Type":"ContainerDied","Data":"5d006c5bd28eb6ccb7dc47c227b6b2f972e651b430df97d6d54dc8803635d9ab"} Sep 11 13:58:14.369385 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:14.369362 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gwb7j" event={"ID":"08e2fadb-d68b-469d-a7f7-6356a9907e6c","Type":"ContainerStarted","Data":"36ebd4c1430a5d5646a0425004677be1ba0212a1c00082be0dad7928870aa0ad"} Sep 11 13:58:14.686484 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:14.686394 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:14.686632 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:14.686553 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 13:58:14.686632 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:14.686627 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls podName:d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:16.686605426 +0000 UTC m=+37.091586538 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls") pod "dns-default-zpsfz" (UID: "d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8") : secret "dns-default-metrics-tls" not found Sep 11 13:58:15.374510 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:15.374475 2566 generic.go:356] "Generic (PLEG): container finished" podID="49578359-6ee4-4d62-94ac-dd9b801007e0" containerID="c6f31193e48b51ffd800abd46d4270e181578bb69dae034e3b82749574c8f559" exitCode=0 Sep 11 13:58:15.375029 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:15.374540 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crfj6" event={"ID":"49578359-6ee4-4d62-94ac-dd9b801007e0","Type":"ContainerDied","Data":"c6f31193e48b51ffd800abd46d4270e181578bb69dae034e3b82749574c8f559"} Sep 11 13:58:16.381276 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:16.381244 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crfj6" event={"ID":"49578359-6ee4-4d62-94ac-dd9b801007e0","Type":"ContainerStarted","Data":"57651d91c64db92b0883ce33cde33f09b5576ebb69b0c96aef202a2656beed2b"} Sep 11 13:58:16.407806 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:16.407762 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-crfj6" podStartSLOduration=7.494624367 podStartE2EDuration="36.40774493s" totalImagesPullingTime="28.913120563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:16.40598245 +0000 UTC m=+36.810963577" watchObservedRunningTime="2026-09-11 13:58:16.40774493 +0000 UTC m=+36.812726059" Sep 11 13:58:16.704867 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:16.704837 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:16.705000 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:16.704981 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 13:58:16.705053 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:16.705045 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls podName:d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:20.705030439 +0000 UTC m=+41.110011545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls") pod "dns-default-zpsfz" (UID: "d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8") : secret "dns-default-metrics-tls" not found Sep 11 13:58:17.385240 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:17.385151 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gwb7j" event={"ID":"08e2fadb-d68b-469d-a7f7-6356a9907e6c","Type":"ContainerStarted","Data":"85d9ecf19816af361c3b7d51feb1cdfd94dd846a5056d01fa6934876547bed3d"} Sep 11 13:58:17.403119 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:17.403057 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gwb7j" podStartSLOduration=2.7837373039999997 podStartE2EDuration="5.403041165s" totalImagesPullingTime="2.619303861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:17.402590617 +0000 UTC m=+37.807571744" watchObservedRunningTime="2026-09-11 13:58:17.403041165 +0000 UTC m=+37.808022293" Sep 11 13:58:20.734553 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:20.734507 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:20.734914 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:20.734630 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 13:58:20.734914 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:20.734689 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls podName:d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:28.734671839 +0000 UTC m=+49.139652956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls") pod "dns-default-zpsfz" (UID: "d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8") : secret "dns-default-metrics-tls" not found Sep 11 13:58:28.787890 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:28.787845 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:28.788306 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:28.787965 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 13:58:28.788306 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:28.788016 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls podName:d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:44.788001309 +0000 UTC m=+65.192982416 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls") pod "dns-default-zpsfz" (UID: "d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8") : secret "dns-default-metrics-tls" not found Sep 11 13:58:38.355543 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:38.355512 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t6zm9" Sep 11 13:58:44.800817 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:44.800759 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:58:44.800817 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:44.800826 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs\") pod \"network-metrics-daemon-hphpv\" (UID: \"98e24790-f943-4aa7-b079-b8940fb02a82\") " pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:58:44.801371 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:44.800851 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:58:44.801371 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:44.800942 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 13:58:44.801371 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:44.800992 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls podName:d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8 nodeName:}" failed. No retries permitted until 2026-09-11 13:59:16.80097828 +0000 UTC m=+97.205959387 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls") pod "dns-default-zpsfz" (UID: "d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8") : secret "dns-default-metrics-tls" not found Sep 11 13:58:44.803733 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:44.803715 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 13:58:44.803793 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:44.803748 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 13:58:44.811453 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:44.811436 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 13:58:44.811508 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:44.811484 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs podName:98e24790-f943-4aa7-b079-b8940fb02a82 nodeName:}" failed. No retries permitted until 2026-09-11 13:59:48.811470561 +0000 UTC m=+129.216451668 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs") pod "network-metrics-daemon-hphpv" (UID: "98e24790-f943-4aa7-b079-b8940fb02a82") : secret "metrics-daemon-secret" not found Sep 11 13:58:44.814538 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:44.814519 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9-original-pull-secret\") pod \"global-pull-secret-syncer-4wl44\" (UID: \"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9\") " pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:58:44.901190 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:44.901137 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8t5m\" (UniqueName: \"kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m\") pod \"network-check-target-snsqs\" (UID: \"ce87902b-b413-4c11-8704-f874a61e1f83\") " pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:58:44.906678 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:44.906660 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 13:58:44.914333 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:44.914312 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4wl44" Sep 11 13:58:44.914874 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:44.914862 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 13:58:44.925333 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:44.925309 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8t5m\" (UniqueName: \"kubernetes.io/projected/ce87902b-b413-4c11-8704-f874a61e1f83-kube-api-access-d8t5m\") pod \"network-check-target-snsqs\" (UID: \"ce87902b-b413-4c11-8704-f874a61e1f83\") " pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:58:45.205812 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:58:45.205788 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b82ace2_23c3_42cc_ae13_6b30c7ebe5b9.slice/crio-3f5d24fabfde7bbb786aa3f26303692b577ef23cdda44304c081350730eaa030 WatchSource:0}: Error finding container 3f5d24fabfde7bbb786aa3f26303692b577ef23cdda44304c081350730eaa030: Status 404 returned error can't find the container with id 3f5d24fabfde7bbb786aa3f26303692b577ef23cdda44304c081350730eaa030 Sep 11 13:58:45.206226 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:45.206207 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4wl44"] Sep 11 13:58:45.207135 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:45.207064 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fgrdt\"" Sep 11 13:58:45.212884 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:45.212865 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:58:45.450197 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:45.450162 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4wl44" event={"ID":"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9","Type":"ContainerStarted","Data":"3f5d24fabfde7bbb786aa3f26303692b577ef23cdda44304c081350730eaa030"} Sep 11 13:58:45.485450 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:45.485426 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-snsqs"] Sep 11 13:58:45.486879 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:58:45.486849 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce87902b_b413_4c11_8704_f874a61e1f83.slice/crio-af3f1646c6d950681845981a03c9606d94c6e5207a12be0af1cd8cdbb43fd4f4 WatchSource:0}: Error finding container af3f1646c6d950681845981a03c9606d94c6e5207a12be0af1cd8cdbb43fd4f4: Status 404 returned error can't find the container with id af3f1646c6d950681845981a03c9606d94c6e5207a12be0af1cd8cdbb43fd4f4 Sep 11 13:58:46.455218 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:46.455141 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-snsqs" event={"ID":"ce87902b-b413-4c11-8704-f874a61e1f83","Type":"ContainerStarted","Data":"af3f1646c6d950681845981a03c9606d94c6e5207a12be0af1cd8cdbb43fd4f4"} Sep 11 13:58:49.313410 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.313375 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-h9mh7"] Sep 11 13:58:49.335833 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.335801 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-h9mh7"] Sep 11 13:58:49.335989 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.335909 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.339262 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.339236 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-m7cq8\"" Sep 11 13:58:49.340189 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.340157 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 11 13:58:49.340189 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.340157 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 13:58:49.340382 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.340187 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 13:58:49.340382 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.340161 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 11 13:58:49.344571 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.344553 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 11 13:58:49.421067 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.421028 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5ghjk"] Sep 11 13:58:49.427475 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.427447 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89a2ef95-796d-4a09-97be-fdaa2d981b30-trusted-ca-bundle\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.427623 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.427501 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv5d9\" (UniqueName: \"kubernetes.io/projected/89a2ef95-796d-4a09-97be-fdaa2d981b30-kube-api-access-vv5d9\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.427623 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.427533 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89a2ef95-796d-4a09-97be-fdaa2d981b30-service-ca-bundle\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.427623 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.427566 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/89a2ef95-796d-4a09-97be-fdaa2d981b30-serving-cert\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.427623 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.427616 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89a2ef95-796d-4a09-97be-fdaa2d981b30-tmp\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.427812 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.427639 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/89a2ef95-796d-4a09-97be-fdaa2d981b30-snapshots\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.443865 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.443836 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-qvnzs"] Sep 11 13:58:49.444029 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.443916 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5ghjk" Sep 11 13:58:49.447223 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.447201 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 11 13:58:49.447339 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.447328 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-s4dqs\"" Sep 11 13:58:49.447455 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.447438 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 11 13:58:49.459790 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.459768 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5ghjk"] Sep 11 13:58:49.459919 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.459809 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-qvnzs"] Sep 11 13:58:49.459967 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.459914 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:49.462530 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.462507 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-6xqlg\"" Sep 11 13:58:49.462728 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.462713 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 11 13:58:49.462936 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.462907 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 11 13:58:49.462936 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.462932 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 11 13:58:49.463071 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.462963 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 11 13:58:49.478667 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.478649 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 11 13:58:49.528794 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.528751 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89a2ef95-796d-4a09-97be-fdaa2d981b30-service-ca-bundle\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.528986 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.528814 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/89a2ef95-796d-4a09-97be-fdaa2d981b30-serving-cert\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.528986 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.528848 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7aadeb-5bf7-48b0-9169-7342cc8469ce-serving-cert\") pod \"console-operator-696884fb9c-qvnzs\" (UID: \"6b7aadeb-5bf7-48b0-9169-7342cc8469ce\") " pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:49.528986 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.528878 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jmrs\" (UniqueName: \"kubernetes.io/projected/6b7aadeb-5bf7-48b0-9169-7342cc8469ce-kube-api-access-6jmrs\") pod \"console-operator-696884fb9c-qvnzs\" (UID: \"6b7aadeb-5bf7-48b0-9169-7342cc8469ce\") " pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:49.528986 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.528958 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89a2ef95-796d-4a09-97be-fdaa2d981b30-tmp\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.529228 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.529029 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/89a2ef95-796d-4a09-97be-fdaa2d981b30-snapshots\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.529228 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.529099 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dmsh\" (UniqueName: \"kubernetes.io/projected/567799ac-c5ee-4fbd-b496-5290693eda3f-kube-api-access-4dmsh\") pod \"volume-data-source-validator-86695b7cc7-5ghjk\" (UID: \"567799ac-c5ee-4fbd-b496-5290693eda3f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5ghjk" Sep 11 13:58:49.529328 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.529299 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b7aadeb-5bf7-48b0-9169-7342cc8469ce-config\") pod \"console-operator-696884fb9c-qvnzs\" (UID: \"6b7aadeb-5bf7-48b0-9169-7342cc8469ce\") " pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:49.529393 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.529336 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89a2ef95-796d-4a09-97be-fdaa2d981b30-tmp\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.529393 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.529355 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89a2ef95-796d-4a09-97be-fdaa2d981b30-trusted-ca-bundle\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.529500 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.529429 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv5d9\" (UniqueName: \"kubernetes.io/projected/89a2ef95-796d-4a09-97be-fdaa2d981b30-kube-api-access-vv5d9\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.529500 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.529459 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b7aadeb-5bf7-48b0-9169-7342cc8469ce-trusted-ca\") pod \"console-operator-696884fb9c-qvnzs\" (UID: \"6b7aadeb-5bf7-48b0-9169-7342cc8469ce\") " pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:49.529635 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.529617 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/89a2ef95-796d-4a09-97be-fdaa2d981b30-snapshots\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.530525 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.530505 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89a2ef95-796d-4a09-97be-fdaa2d981b30-trusted-ca-bundle\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.531875 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.531854 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/89a2ef95-796d-4a09-97be-fdaa2d981b30-serving-cert\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.538359 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.538339 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv5d9\" (UniqueName: \"kubernetes.io/projected/89a2ef95-796d-4a09-97be-fdaa2d981b30-kube-api-access-vv5d9\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.542334 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.542312 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89a2ef95-796d-4a09-97be-fdaa2d981b30-service-ca-bundle\") pod \"insights-operator-5844b856db-h9mh7\" (UID: \"89a2ef95-796d-4a09-97be-fdaa2d981b30\") " pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.630631 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.630546 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dmsh\" (UniqueName: \"kubernetes.io/projected/567799ac-c5ee-4fbd-b496-5290693eda3f-kube-api-access-4dmsh\") pod \"volume-data-source-validator-86695b7cc7-5ghjk\" (UID: \"567799ac-c5ee-4fbd-b496-5290693eda3f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5ghjk" Sep 11 13:58:49.630631 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.630605 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b7aadeb-5bf7-48b0-9169-7342cc8469ce-config\") pod \"console-operator-696884fb9c-qvnzs\" (UID: \"6b7aadeb-5bf7-48b0-9169-7342cc8469ce\") " pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:49.630845 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.630665 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b7aadeb-5bf7-48b0-9169-7342cc8469ce-trusted-ca\") pod \"console-operator-696884fb9c-qvnzs\" (UID: \"6b7aadeb-5bf7-48b0-9169-7342cc8469ce\") " pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:49.630953 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.630932 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7aadeb-5bf7-48b0-9169-7342cc8469ce-serving-cert\") pod \"console-operator-696884fb9c-qvnzs\" (UID: \"6b7aadeb-5bf7-48b0-9169-7342cc8469ce\") " pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:49.631036 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.630966 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmrs\" (UniqueName: \"kubernetes.io/projected/6b7aadeb-5bf7-48b0-9169-7342cc8469ce-kube-api-access-6jmrs\") pod \"console-operator-696884fb9c-qvnzs\" (UID: \"6b7aadeb-5bf7-48b0-9169-7342cc8469ce\") " pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:49.631397 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.631360 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b7aadeb-5bf7-48b0-9169-7342cc8469ce-config\") pod \"console-operator-696884fb9c-qvnzs\" (UID: \"6b7aadeb-5bf7-48b0-9169-7342cc8469ce\") " pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:49.631632 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.631612 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b7aadeb-5bf7-48b0-9169-7342cc8469ce-trusted-ca\") pod \"console-operator-696884fb9c-qvnzs\" (UID: \"6b7aadeb-5bf7-48b0-9169-7342cc8469ce\") " pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:49.633570 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.633547 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7aadeb-5bf7-48b0-9169-7342cc8469ce-serving-cert\") pod \"console-operator-696884fb9c-qvnzs\" (UID: \"6b7aadeb-5bf7-48b0-9169-7342cc8469ce\") " pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:49.639324 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.639300 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dmsh\" (UniqueName: \"kubernetes.io/projected/567799ac-c5ee-4fbd-b496-5290693eda3f-kube-api-access-4dmsh\") pod \"volume-data-source-validator-86695b7cc7-5ghjk\" (UID: \"567799ac-c5ee-4fbd-b496-5290693eda3f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5ghjk" Sep 11 13:58:49.639324 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.639317 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmrs\" (UniqueName: \"kubernetes.io/projected/6b7aadeb-5bf7-48b0-9169-7342cc8469ce-kube-api-access-6jmrs\") pod \"console-operator-696884fb9c-qvnzs\" (UID: \"6b7aadeb-5bf7-48b0-9169-7342cc8469ce\") " pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:49.647108 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.647075 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-h9mh7" Sep 11 13:58:49.755013 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.754973 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5ghjk" Sep 11 13:58:49.770368 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:49.770337 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:50.307234 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.306913 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-qvnzs"] Sep 11 13:58:50.307931 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.307905 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5ghjk"] Sep 11 13:58:50.309513 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:58:50.309478 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7aadeb_5bf7_48b0_9169_7342cc8469ce.slice/crio-860d6edd8dbc47ac34a1815375954458c8408979cdd6627ed1a4382a961ed848 WatchSource:0}: Error finding container 860d6edd8dbc47ac34a1815375954458c8408979cdd6627ed1a4382a961ed848: Status 404 returned error can't find the container with id 860d6edd8dbc47ac34a1815375954458c8408979cdd6627ed1a4382a961ed848 Sep 11 13:58:50.310041 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:58:50.310022 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567799ac_c5ee_4fbd_b496_5290693eda3f.slice/crio-a2d883317146bcea87bc31fbea1f5609391ce308cf843dc32933db23f64a16a6 WatchSource:0}: Error finding container a2d883317146bcea87bc31fbea1f5609391ce308cf843dc32933db23f64a16a6: Status 404 returned error can't find the container with id a2d883317146bcea87bc31fbea1f5609391ce308cf843dc32933db23f64a16a6 Sep 11 13:58:50.323946 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.323922 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-h9mh7"] Sep 11 13:58:50.327528 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:58:50.327503 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a2ef95_796d_4a09_97be_fdaa2d981b30.slice/crio-1316cef8278c1958cc039f76d7ebf48a152618e9fd3dee8ec83deb514291c3e0 WatchSource:0}: Error finding container 1316cef8278c1958cc039f76d7ebf48a152618e9fd3dee8ec83deb514291c3e0: Status 404 returned error can't find the container with id 1316cef8278c1958cc039f76d7ebf48a152618e9fd3dee8ec83deb514291c3e0 Sep 11 13:58:50.467419 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.467378 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4wl44" event={"ID":"3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9","Type":"ContainerStarted","Data":"faf41df95627b628097aaead7b8f4cd8bb73a7443472226e400f1c59471a566d"} Sep 11 13:58:50.468831 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.468798 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-snsqs" event={"ID":"ce87902b-b413-4c11-8704-f874a61e1f83","Type":"ContainerStarted","Data":"8ffe22c50df871acc6146ea32f71bd68442e9ce42695acf38cdc506cbc2ab638"} Sep 11 13:58:50.468973 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.468872 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:58:50.469935 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.469913 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h9mh7" event={"ID":"89a2ef95-796d-4a09-97be-fdaa2d981b30","Type":"ContainerStarted","Data":"1316cef8278c1958cc039f76d7ebf48a152618e9fd3dee8ec83deb514291c3e0"} Sep 11 13:58:50.471223 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.471185 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" event={"ID":"6b7aadeb-5bf7-48b0-9169-7342cc8469ce","Type":"ContainerStarted","Data":"860d6edd8dbc47ac34a1815375954458c8408979cdd6627ed1a4382a961ed848"} Sep 11 13:58:50.472330 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.472313 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5ghjk" event={"ID":"567799ac-c5ee-4fbd-b496-5290693eda3f","Type":"ContainerStarted","Data":"a2d883317146bcea87bc31fbea1f5609391ce308cf843dc32933db23f64a16a6"} Sep 11 13:58:50.484745 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.484704 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4wl44" podStartSLOduration=65.65675627 podStartE2EDuration="1m10.484691361s" totalImagesPullingTime="4.827935091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:50.483928399 +0000 UTC m=+70.888909527" watchObservedRunningTime="2026-09-11 13:58:50.484691361 +0000 UTC m=+70.889672482" Sep 11 13:58:50.518880 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.518829 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-snsqs" podStartSLOduration=65.987427888 podStartE2EDuration="1m10.518813886s" totalImagesPullingTime="4.531385998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:50.518058699 +0000 UTC m=+70.923039818" watchObservedRunningTime="2026-09-11 13:58:50.518813886 +0000 UTC m=+70.923795011" Sep 11 13:58:50.963525 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.963494 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59c887d6c5-wm7zl"] Sep 11 13:58:50.965322 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.965293 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:50.969714 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.969690 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 13:58:50.969850 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.969765 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kzr4h\"" Sep 11 13:58:50.970459 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.970437 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 13:58:50.970594 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.970461 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 13:58:50.976817 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.976776 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 13:58:50.979484 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:50.979464 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59c887d6c5-wm7zl"] Sep 11 13:58:51.042212 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.042175 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d70e5b7a-dfee-4881-a8fb-24f858af429e-image-registry-private-configuration\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.042212 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.042224 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-742js\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-kube-api-access-742js\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.042451 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.042263 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.042451 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.042290 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d70e5b7a-dfee-4881-a8fb-24f858af429e-ca-trust-extracted\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.042451 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.042316 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-certificates\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.042451 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.042382 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d70e5b7a-dfee-4881-a8fb-24f858af429e-installation-pull-secrets\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.042451 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.042417 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d70e5b7a-dfee-4881-a8fb-24f858af429e-trusted-ca\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.042451 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.042439 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-bound-sa-token\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.145104 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.143552 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d70e5b7a-dfee-4881-a8fb-24f858af429e-image-registry-private-configuration\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.145104 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.143612 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-742js\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-kube-api-access-742js\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.145104 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.143650 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.145104 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.143677 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d70e5b7a-dfee-4881-a8fb-24f858af429e-ca-trust-extracted\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.145104 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.143703 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-certificates\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.145104 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.143766 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d70e5b7a-dfee-4881-a8fb-24f858af429e-installation-pull-secrets\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.145104 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.143804 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d70e5b7a-dfee-4881-a8fb-24f858af429e-trusted-ca\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.145104 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.143825 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-bound-sa-token\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.145104 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:51.144713 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 13:58:51.145104 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:51.144737 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c887d6c5-wm7zl: secret "image-registry-tls" not found Sep 11 13:58:51.145104 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:51.144804 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls podName:d70e5b7a-dfee-4881-a8fb-24f858af429e nodeName:}" failed. No retries permitted until 2026-09-11 13:58:51.644775095 +0000 UTC m=+72.049756217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls") pod "image-registry-59c887d6c5-wm7zl" (UID: "d70e5b7a-dfee-4881-a8fb-24f858af429e") : secret "image-registry-tls" not found Sep 11 13:58:51.145104 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.144906 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d70e5b7a-dfee-4881-a8fb-24f858af429e-ca-trust-extracted\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.147702 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.147671 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d70e5b7a-dfee-4881-a8fb-24f858af429e-trusted-ca\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.147982 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.147958 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-certificates\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.151675 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.149101 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d70e5b7a-dfee-4881-a8fb-24f858af429e-image-registry-private-configuration\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.152564 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.152487 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d70e5b7a-dfee-4881-a8fb-24f858af429e-installation-pull-secrets\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.164312 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.164255 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-bound-sa-token\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.164949 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.164926 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-742js\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-kube-api-access-742js\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.481054 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.481011 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5ghjk" event={"ID":"567799ac-c5ee-4fbd-b496-5290693eda3f","Type":"ContainerStarted","Data":"6245f570349f12acd49e49393f87a44b85aa05b9aa55d47ab75b72d123554f47"} Sep 11 13:58:51.498114 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.497664 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5ghjk" podStartSLOduration=1.493860088 podStartE2EDuration="2.497648975s" totalImagesPullingTime="1.003788887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:51.497237026 +0000 UTC m=+71.902218157" watchObservedRunningTime="2026-09-11 13:58:51.497648975 +0000 UTC m=+71.902630103" Sep 11 13:58:51.648744 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:51.648648 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:51.648919 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:51.648840 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 13:58:51.648919 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:51.648863 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c887d6c5-wm7zl: secret "image-registry-tls" not found Sep 11 13:58:51.649000 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:51.648938 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls podName:d70e5b7a-dfee-4881-a8fb-24f858af429e nodeName:}" failed. No retries permitted until 2026-09-11 13:58:52.648914847 +0000 UTC m=+73.053895967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls") pod "image-registry-59c887d6c5-wm7zl" (UID: "d70e5b7a-dfee-4881-a8fb-24f858af429e") : secret "image-registry-tls" not found Sep 11 13:58:52.657965 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:52.657925 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:52.658416 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:52.658123 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 13:58:52.658416 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:52.658150 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c887d6c5-wm7zl: secret "image-registry-tls" not found Sep 11 13:58:52.658416 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:52.658231 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls podName:d70e5b7a-dfee-4881-a8fb-24f858af429e nodeName:}" failed. No retries permitted until 2026-09-11 13:58:54.658213391 +0000 UTC m=+75.063194501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls") pod "image-registry-59c887d6c5-wm7zl" (UID: "d70e5b7a-dfee-4881-a8fb-24f858af429e") : secret "image-registry-tls" not found Sep 11 13:58:53.489160 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:53.489110 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h9mh7" event={"ID":"89a2ef95-796d-4a09-97be-fdaa2d981b30","Type":"ContainerStarted","Data":"5b46455c47655119b757787868ee567ec8e380216c5cfcd7001e980d60a70f5d"} Sep 11 13:58:53.490893 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:53.490870 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/0.log" Sep 11 13:58:53.491008 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:53.490915 2566 generic.go:356] "Generic (PLEG): container finished" podID="6b7aadeb-5bf7-48b0-9169-7342cc8469ce" containerID="fa09d20861a4904db42a8558030728a0cdcea806ef7bb69868136e1825811c5c" exitCode=255 Sep 11 13:58:53.491008 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:53.490959 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" event={"ID":"6b7aadeb-5bf7-48b0-9169-7342cc8469ce","Type":"ContainerDied","Data":"fa09d20861a4904db42a8558030728a0cdcea806ef7bb69868136e1825811c5c"} Sep 11 13:58:53.491212 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:53.491199 2566 scope.go:117] "RemoveContainer" containerID="fa09d20861a4904db42a8558030728a0cdcea806ef7bb69868136e1825811c5c" Sep 11 13:58:53.508149 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:53.508107 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-h9mh7" podStartSLOduration=2.028746139 podStartE2EDuration="4.50806955s" totalImagesPullingTime="2.479323411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:53.507122437 +0000 UTC m=+73.912103565" watchObservedRunningTime="2026-09-11 13:58:53.50806955 +0000 UTC m=+73.913050678" Sep 11 13:58:54.328532 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.328495 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vkpjz"] Sep 11 13:58:54.331505 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.331486 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.334485 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.334468 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-85l57\"" Sep 11 13:58:54.334560 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.334517 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 13:58:54.341367 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.341337 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 13:58:54.350684 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.350657 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vkpjz"] Sep 11 13:58:54.370890 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.370854 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba31e115-ed2c-4929-8c37-d4342d62147f-data-volume\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.371055 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.370894 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba31e115-ed2c-4929-8c37-d4342d62147f-crio-socket\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.371055 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.370934 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg5m2\" (UniqueName: \"kubernetes.io/projected/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-api-access-hg5m2\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.371055 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.370993 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.371212 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.371189 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.471986 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.471939 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.472183 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.472003 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba31e115-ed2c-4929-8c37-d4342d62147f-data-volume\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.472183 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.472028 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba31e115-ed2c-4929-8c37-d4342d62147f-crio-socket\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.472183 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.472071 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg5m2\" (UniqueName: \"kubernetes.io/projected/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-api-access-hg5m2\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.472183 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.472121 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.472183 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:54.472122 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 13:58:54.472183 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:54.472169 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 13:58:54.472424 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.472192 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba31e115-ed2c-4929-8c37-d4342d62147f-crio-socket\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.472424 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:54.472244 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm podName:ba31e115-ed2c-4929-8c37-d4342d62147f nodeName:}" failed. No retries permitted until 2026-09-11 13:58:54.972224043 +0000 UTC m=+75.377205164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vkpjz" (UID: "ba31e115-ed2c-4929-8c37-d4342d62147f") : configmap "kube-rbac-proxy" not found Sep 11 13:58:54.472424 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:54.472299 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls podName:ba31e115-ed2c-4929-8c37-d4342d62147f nodeName:}" failed. No retries permitted until 2026-09-11 13:58:54.972254046 +0000 UTC m=+75.377235165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkpjz" (UID: "ba31e115-ed2c-4929-8c37-d4342d62147f") : secret "insights-runtime-extractor-tls" not found Sep 11 13:58:54.472529 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.472431 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba31e115-ed2c-4929-8c37-d4342d62147f-data-volume\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.480958 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.480929 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg5m2\" (UniqueName: \"kubernetes.io/projected/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-api-access-hg5m2\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.495283 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.495264 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/1.log" Sep 11 13:58:54.495676 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.495663 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/0.log" Sep 11 13:58:54.495726 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.495695 2566 generic.go:356] "Generic (PLEG): container finished" podID="6b7aadeb-5bf7-48b0-9169-7342cc8469ce" containerID="2a31cd18fe582e22984c7f92161876f7ce2843459d8cf93c5ef444a5e5c8bc53" exitCode=255 Sep 11 13:58:54.495817 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.495787 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" event={"ID":"6b7aadeb-5bf7-48b0-9169-7342cc8469ce","Type":"ContainerDied","Data":"2a31cd18fe582e22984c7f92161876f7ce2843459d8cf93c5ef444a5e5c8bc53"} Sep 11 13:58:54.495929 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.495846 2566 scope.go:117] "RemoveContainer" containerID="fa09d20861a4904db42a8558030728a0cdcea806ef7bb69868136e1825811c5c" Sep 11 13:58:54.495992 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.495976 2566 scope.go:117] "RemoveContainer" containerID="2a31cd18fe582e22984c7f92161876f7ce2843459d8cf93c5ef444a5e5c8bc53" Sep 11 13:58:54.496198 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:54.496179 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-qvnzs_openshift-console-operator(6b7aadeb-5bf7-48b0-9169-7342cc8469ce)\"" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" podUID="6b7aadeb-5bf7-48b0-9169-7342cc8469ce" Sep 11 13:58:54.673727 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.673644 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:54.673876 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:54.673779 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 13:58:54.673876 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:54.673792 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c887d6c5-wm7zl: secret "image-registry-tls" not found Sep 11 13:58:54.673876 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:54.673842 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls podName:d70e5b7a-dfee-4881-a8fb-24f858af429e nodeName:}" failed. No retries permitted until 2026-09-11 13:58:58.673829379 +0000 UTC m=+79.078810489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls") pod "image-registry-59c887d6c5-wm7zl" (UID: "d70e5b7a-dfee-4881-a8fb-24f858af429e") : secret "image-registry-tls" not found Sep 11 13:58:54.976407 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.976373 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.976566 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:54.976443 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:54.976566 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:54.976522 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 13:58:54.976566 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:54.976526 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 13:58:54.976696 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:54.976595 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls podName:ba31e115-ed2c-4929-8c37-d4342d62147f nodeName:}" failed. No retries permitted until 2026-09-11 13:58:55.97657986 +0000 UTC m=+76.381560967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkpjz" (UID: "ba31e115-ed2c-4929-8c37-d4342d62147f") : secret "insights-runtime-extractor-tls" not found Sep 11 13:58:54.976696 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:54.976610 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm podName:ba31e115-ed2c-4929-8c37-d4342d62147f nodeName:}" failed. No retries permitted until 2026-09-11 13:58:55.976603124 +0000 UTC m=+76.381584230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vkpjz" (UID: "ba31e115-ed2c-4929-8c37-d4342d62147f") : configmap "kube-rbac-proxy" not found Sep 11 13:58:55.500840 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:55.500814 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/1.log" Sep 11 13:58:55.501231 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:55.501164 2566 scope.go:117] "RemoveContainer" containerID="2a31cd18fe582e22984c7f92161876f7ce2843459d8cf93c5ef444a5e5c8bc53" Sep 11 13:58:55.501324 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:55.501305 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-qvnzs_openshift-console-operator(6b7aadeb-5bf7-48b0-9169-7342cc8469ce)\"" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" podUID="6b7aadeb-5bf7-48b0-9169-7342cc8469ce" Sep 11 13:58:55.985195 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:55.985169 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:55.985354 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:55.985223 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:55.985354 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:55.985300 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 13:58:55.985354 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:55.985328 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 13:58:55.985451 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:55.985365 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm podName:ba31e115-ed2c-4929-8c37-d4342d62147f nodeName:}" failed. No retries permitted until 2026-09-11 13:58:57.985346997 +0000 UTC m=+78.390328126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vkpjz" (UID: "ba31e115-ed2c-4929-8c37-d4342d62147f") : configmap "kube-rbac-proxy" not found Sep 11 13:58:55.985451 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:55.985380 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls podName:ba31e115-ed2c-4929-8c37-d4342d62147f nodeName:}" failed. No retries permitted until 2026-09-11 13:58:57.985374308 +0000 UTC m=+78.390355414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkpjz" (UID: "ba31e115-ed2c-4929-8c37-d4342d62147f") : secret "insights-runtime-extractor-tls" not found Sep 11 13:58:56.426171 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:56.426070 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pm6vt_0ffa4002-eab4-4d53-aad1-c82ba3810441/dns-node-resolver/0.log" Sep 11 13:58:57.425017 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:57.424986 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mjmwj_c2c22dff-0209-4bda-9a7a-5626d2268974/node-ca/0.log" Sep 11 13:58:57.825063 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:57.825031 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gwb7j_08e2fadb-d68b-469d-a7f7-6356a9907e6c/serve-healthcheck-canary/0.log" Sep 11 13:58:57.999233 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:57.999197 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:57.999396 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:57.999262 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:58:57.999396 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:57.999323 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 13:58:57.999396 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:57.999365 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 13:58:57.999507 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:57.999398 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm podName:ba31e115-ed2c-4929-8c37-d4342d62147f nodeName:}" failed. No retries permitted until 2026-09-11 13:59:01.999380438 +0000 UTC m=+82.404361544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vkpjz" (UID: "ba31e115-ed2c-4929-8c37-d4342d62147f") : configmap "kube-rbac-proxy" not found Sep 11 13:58:57.999507 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:57.999411 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls podName:ba31e115-ed2c-4929-8c37-d4342d62147f nodeName:}" failed. No retries permitted until 2026-09-11 13:59:01.999405708 +0000 UTC m=+82.404386814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkpjz" (UID: "ba31e115-ed2c-4929-8c37-d4342d62147f") : secret "insights-runtime-extractor-tls" not found Sep 11 13:58:58.705117 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:58.705053 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:58:58.705508 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:58.705205 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 13:58:58.705508 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:58.705230 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c887d6c5-wm7zl: secret "image-registry-tls" not found Sep 11 13:58:58.705508 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:58.705304 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls podName:d70e5b7a-dfee-4881-a8fb-24f858af429e nodeName:}" failed. No retries permitted until 2026-09-11 13:59:06.705287323 +0000 UTC m=+87.110268429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls") pod "image-registry-59c887d6c5-wm7zl" (UID: "d70e5b7a-dfee-4881-a8fb-24f858af429e") : secret "image-registry-tls" not found Sep 11 13:58:59.771127 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:59.771076 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:59.771530 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:59.771154 2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:58:59.771530 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:58:59.771452 2566 scope.go:117] "RemoveContainer" containerID="2a31cd18fe582e22984c7f92161876f7ce2843459d8cf93c5ef444a5e5c8bc53" Sep 11 13:58:59.771613 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:58:59.771595 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-qvnzs_openshift-console-operator(6b7aadeb-5bf7-48b0-9169-7342cc8469ce)\"" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" podUID="6b7aadeb-5bf7-48b0-9169-7342cc8469ce" Sep 11 13:59:02.029912 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:02.029798 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:59:02.029912 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:02.029864 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:59:02.030317 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:02.029927 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 13:59:02.030317 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:02.029988 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 13:59:02.030317 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:02.030003 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm podName:ba31e115-ed2c-4929-8c37-d4342d62147f nodeName:}" failed. No retries permitted until 2026-09-11 13:59:10.029987406 +0000 UTC m=+90.434968511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vkpjz" (UID: "ba31e115-ed2c-4929-8c37-d4342d62147f") : configmap "kube-rbac-proxy" not found Sep 11 13:59:02.030317 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:02.030026 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls podName:ba31e115-ed2c-4929-8c37-d4342d62147f nodeName:}" failed. No retries permitted until 2026-09-11 13:59:10.030015212 +0000 UTC m=+90.434996318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkpjz" (UID: "ba31e115-ed2c-4929-8c37-d4342d62147f") : secret "insights-runtime-extractor-tls" not found Sep 11 13:59:06.767373 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:06.767329 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:59:06.767723 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:06.767478 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 13:59:06.767723 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:06.767494 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c887d6c5-wm7zl: secret "image-registry-tls" not found Sep 11 13:59:06.767723 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:06.767566 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls podName:d70e5b7a-dfee-4881-a8fb-24f858af429e nodeName:}" failed. No retries permitted until 2026-09-11 13:59:22.767548271 +0000 UTC m=+103.172529394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls") pod "image-registry-59c887d6c5-wm7zl" (UID: "d70e5b7a-dfee-4881-a8fb-24f858af429e") : secret "image-registry-tls" not found Sep 11 13:59:10.091485 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:10.091447 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:59:10.091875 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:10.091513 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:59:10.091875 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:10.091639 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 13:59:10.091875 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:10.091733 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls podName:ba31e115-ed2c-4929-8c37-d4342d62147f nodeName:}" failed. No retries permitted until 2026-09-11 13:59:26.091711354 +0000 UTC m=+106.496692474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vkpjz" (UID: "ba31e115-ed2c-4929-8c37-d4342d62147f") : secret "insights-runtime-extractor-tls" not found Sep 11 13:59:10.092002 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:10.091985 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba31e115-ed2c-4929-8c37-d4342d62147f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:59:10.092802 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:10.092785 2566 scope.go:117] "RemoveContainer" containerID="2a31cd18fe582e22984c7f92161876f7ce2843459d8cf93c5ef444a5e5c8bc53" Sep 11 13:59:10.551053 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:10.551024 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 13:59:10.551422 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:10.551405 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/1.log" Sep 11 13:59:10.551524 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:10.551445 2566 generic.go:356] "Generic (PLEG): container finished" podID="6b7aadeb-5bf7-48b0-9169-7342cc8469ce" containerID="a59598c8a5dafb266e31f3d97cefea999145ef380d9a4d52e99fff2e802330ef" exitCode=255 Sep 11 13:59:10.551524 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:10.551504 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" event={"ID":"6b7aadeb-5bf7-48b0-9169-7342cc8469ce","Type":"ContainerDied","Data":"a59598c8a5dafb266e31f3d97cefea999145ef380d9a4d52e99fff2e802330ef"} Sep 11 13:59:10.551684 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:10.551600 2566 scope.go:117] "RemoveContainer" containerID="2a31cd18fe582e22984c7f92161876f7ce2843459d8cf93c5ef444a5e5c8bc53" Sep 11 13:59:10.551929 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:10.551912 2566 scope.go:117] "RemoveContainer" containerID="a59598c8a5dafb266e31f3d97cefea999145ef380d9a4d52e99fff2e802330ef" Sep 11 13:59:10.552105 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:10.552072 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-qvnzs_openshift-console-operator(6b7aadeb-5bf7-48b0-9169-7342cc8469ce)\"" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" podUID="6b7aadeb-5bf7-48b0-9169-7342cc8469ce" Sep 11 13:59:11.556134 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:11.556108 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 13:59:16.843960 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:16.843911 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:59:16.846330 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:16.846298 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8-metrics-tls\") pod \"dns-default-zpsfz\" (UID: \"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8\") " pod="openshift-dns/dns-default-zpsfz" Sep 11 13:59:16.919990 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:16.919959 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tbm52\"" Sep 11 13:59:16.928185 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:16.928167 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zpsfz" Sep 11 13:59:17.214495 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:17.214467 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zpsfz"] Sep 11 13:59:17.217688 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:59:17.217659 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6aa3fd6_aab7_4feb_8f3c_3762e7e1c9d8.slice/crio-42637f987ee0efc13636a78d79ba40b6315f2d8ac6408b16a4702f4a37dc30c2 WatchSource:0}: Error finding container 42637f987ee0efc13636a78d79ba40b6315f2d8ac6408b16a4702f4a37dc30c2: Status 404 returned error can't find the container with id 42637f987ee0efc13636a78d79ba40b6315f2d8ac6408b16a4702f4a37dc30c2 Sep 11 13:59:17.574776 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:17.574736 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zpsfz" event={"ID":"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8","Type":"ContainerStarted","Data":"42637f987ee0efc13636a78d79ba40b6315f2d8ac6408b16a4702f4a37dc30c2"} Sep 11 13:59:19.582909 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:19.582870 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zpsfz" event={"ID":"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8","Type":"ContainerStarted","Data":"8fda8caad4d8ac4b41ab1566aab88df7eecb68efa6a882f1a51779d1c447476e"} Sep 11 13:59:19.583315 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:19.582932 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zpsfz" event={"ID":"d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8","Type":"ContainerStarted","Data":"d15b4262df43f68a4e860c6d01d732f1a98aac2d9bc7cb0b3a2d6287ba5099f4"} Sep 11 13:59:19.583315 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:19.582990 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zpsfz" Sep 11 13:59:19.601195 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:19.601145 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zpsfz" podStartSLOduration=65.74603781 podStartE2EDuration="1m7.601128323s" totalImagesPullingTime="1.855090513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:19.599775624 +0000 UTC m=+100.004756744" watchObservedRunningTime="2026-09-11 13:59:19.601128323 +0000 UTC m=+100.006109451" Sep 11 13:59:19.770425 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:19.770392 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:59:19.770622 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:19.770488 2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:59:19.770729 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:19.770716 2566 scope.go:117] "RemoveContainer" containerID="a59598c8a5dafb266e31f3d97cefea999145ef380d9a4d52e99fff2e802330ef" Sep 11 13:59:19.770872 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:19.770855 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-qvnzs_openshift-console-operator(6b7aadeb-5bf7-48b0-9169-7342cc8469ce)\"" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" podUID="6b7aadeb-5bf7-48b0-9169-7342cc8469ce" Sep 11 13:59:20.585956 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:20.585929 2566 scope.go:117] "RemoveContainer" containerID="a59598c8a5dafb266e31f3d97cefea999145ef380d9a4d52e99fff2e802330ef" Sep 11 13:59:20.586376 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:20.586137 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-qvnzs_openshift-console-operator(6b7aadeb-5bf7-48b0-9169-7342cc8469ce)\"" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" podUID="6b7aadeb-5bf7-48b0-9169-7342cc8469ce" Sep 11 13:59:21.483344 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:21.483312 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-snsqs" Sep 11 13:59:22.792719 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:22.792679 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:59:22.795341 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:22.795320 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls\") pod \"image-registry-59c887d6c5-wm7zl\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:59:23.080462 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:23.080378 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:59:23.379449 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:23.379361 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59c887d6c5-wm7zl"] Sep 11 13:59:23.381916 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:59:23.381892 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70e5b7a_dfee_4881_a8fb_24f858af429e.slice/crio-1c48e54900b5b1cbb7de2eb56cdff7f4c32a769a716895d791d9243e4fc0b8df WatchSource:0}: Error finding container 1c48e54900b5b1cbb7de2eb56cdff7f4c32a769a716895d791d9243e4fc0b8df: Status 404 returned error can't find the container with id 1c48e54900b5b1cbb7de2eb56cdff7f4c32a769a716895d791d9243e4fc0b8df Sep 11 13:59:23.598405 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:23.598369 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" event={"ID":"d70e5b7a-dfee-4881-a8fb-24f858af429e","Type":"ContainerStarted","Data":"520b6d66fa5bf03e43555274337f16a8bf6bea92aecdb5376c246c61ba7f28e8"} Sep 11 13:59:23.598583 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:23.598428 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" event={"ID":"d70e5b7a-dfee-4881-a8fb-24f858af429e","Type":"ContainerStarted","Data":"1c48e54900b5b1cbb7de2eb56cdff7f4c32a769a716895d791d9243e4fc0b8df"} Sep 11 13:59:23.598583 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:23.598536 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:59:23.620245 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:23.620013 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" podStartSLOduration=33.619991398 podStartE2EDuration="33.619991398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:23.619678844 +0000 UTC m=+104.024659973" watchObservedRunningTime="2026-09-11 13:59:23.619991398 +0000 UTC m=+104.024972526" Sep 11 13:59:26.117055 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:26.117019 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:59:26.119430 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:26.119403 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba31e115-ed2c-4929-8c37-d4342d62147f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vkpjz\" (UID: \"ba31e115-ed2c-4929-8c37-d4342d62147f\") " pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:59:26.139506 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:26.139477 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vkpjz" Sep 11 13:59:26.450337 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:26.450251 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vkpjz"] Sep 11 13:59:26.453251 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:59:26.453227 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba31e115_ed2c_4929_8c37_d4342d62147f.slice/crio-1f2a04b3458f142115987dac8062e8deff547c828258787499e490cd95b301fd WatchSource:0}: Error finding container 1f2a04b3458f142115987dac8062e8deff547c828258787499e490cd95b301fd: Status 404 returned error can't find the container with id 1f2a04b3458f142115987dac8062e8deff547c828258787499e490cd95b301fd Sep 11 13:59:26.609669 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:26.609638 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkpjz" event={"ID":"ba31e115-ed2c-4929-8c37-d4342d62147f","Type":"ContainerStarted","Data":"c5b4b71869a8019f5e7dcf794adec228e92a7472b2fdce8b3b37c51306a41991"} Sep 11 13:59:26.609809 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:26.609694 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkpjz" event={"ID":"ba31e115-ed2c-4929-8c37-d4342d62147f","Type":"ContainerStarted","Data":"1f2a04b3458f142115987dac8062e8deff547c828258787499e490cd95b301fd"} Sep 11 13:59:27.614680 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:27.614639 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkpjz" event={"ID":"ba31e115-ed2c-4929-8c37-d4342d62147f","Type":"ContainerStarted","Data":"ce212d27ce4542c722604547957b7069739aa8851ba55a94ea402d4a5990c11e"} Sep 11 13:59:27.957693 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:27.957620 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mlgdz"] Sep 11 13:59:27.959397 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:27.959381 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mlgdz" Sep 11 13:59:27.962945 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:27.962923 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 11 13:59:27.963031 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:27.963008 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tpjt5\"" Sep 11 13:59:27.975597 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:27.975573 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59c887d6c5-wm7zl"] Sep 11 13:59:27.976723 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:27.976696 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mlgdz"] Sep 11 13:59:28.031045 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:28.031004 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f810a603-3b8d-498a-9b13-dcf15aed909e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mlgdz\" (UID: \"f810a603-3b8d-498a-9b13-dcf15aed909e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mlgdz" Sep 11 13:59:28.131627 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:28.131584 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f810a603-3b8d-498a-9b13-dcf15aed909e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mlgdz\" (UID: \"f810a603-3b8d-498a-9b13-dcf15aed909e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mlgdz" Sep 11 13:59:28.134112 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:28.134076 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f810a603-3b8d-498a-9b13-dcf15aed909e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-mlgdz\" (UID: \"f810a603-3b8d-498a-9b13-dcf15aed909e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mlgdz" Sep 11 13:59:28.268342 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:28.268306 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mlgdz" Sep 11 13:59:28.546061 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:28.545987 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mlgdz"] Sep 11 13:59:28.754789 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:59:28.754758 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf810a603_3b8d_498a_9b13_dcf15aed909e.slice/crio-358731b7df391f87a90d544710cbb06391df5cc711a6c5771ddbd7506c6b9701 WatchSource:0}: Error finding container 358731b7df391f87a90d544710cbb06391df5cc711a6c5771ddbd7506c6b9701: Status 404 returned error can't find the container with id 358731b7df391f87a90d544710cbb06391df5cc711a6c5771ddbd7506c6b9701 Sep 11 13:59:29.588171 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:29.588141 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zpsfz" Sep 11 13:59:29.625462 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:29.624904 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vkpjz" event={"ID":"ba31e115-ed2c-4929-8c37-d4342d62147f","Type":"ContainerStarted","Data":"96b45353e2dc9765a0bcf7f4593f9462c368f2042f22ff02d8881b552ddda7cc"} Sep 11 13:59:29.626521 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:29.626492 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mlgdz" event={"ID":"f810a603-3b8d-498a-9b13-dcf15aed909e","Type":"ContainerStarted","Data":"358731b7df391f87a90d544710cbb06391df5cc711a6c5771ddbd7506c6b9701"} Sep 11 13:59:29.649676 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:29.649618 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vkpjz" podStartSLOduration=33.507706153 podStartE2EDuration="35.649599763s" totalImagesPullingTime="2.14189361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:58:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:29.649035727 +0000 UTC m=+110.054016860" watchObservedRunningTime="2026-09-11 13:59:29.649599763 +0000 UTC m=+110.054580895" Sep 11 13:59:30.631059 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:30.631019 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mlgdz" event={"ID":"f810a603-3b8d-498a-9b13-dcf15aed909e","Type":"ContainerStarted","Data":"ce06a2f09230c1cc2f24d547e85d0fd358f01e393bf1ba836d6414fa2acb9483"} Sep 11 13:59:30.631568 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:30.631120 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mlgdz" Sep 11 13:59:30.635771 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:30.635745 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mlgdz" Sep 11 13:59:30.647132 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:30.647069 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-mlgdz" podStartSLOduration=2.584547539 podStartE2EDuration="3.647056484s" totalImagesPullingTime="1.062508945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:30.646299576 +0000 UTC m=+111.051280695" watchObservedRunningTime="2026-09-11 13:59:30.647056484 +0000 UTC m=+111.052037611" Sep 11 13:59:31.091847 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.091812 2566 scope.go:117] "RemoveContainer" containerID="a59598c8a5dafb266e31f3d97cefea999145ef380d9a4d52e99fff2e802330ef" Sep 11 13:59:31.553406 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.553369 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2"] Sep 11 13:59:31.555184 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.555168 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" Sep 11 13:59:31.558976 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.558939 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 11 13:59:31.558976 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.558944 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 13:59:31.558976 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.558970 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 13:59:31.558976 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.558979 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5fc5p\"" Sep 11 13:59:31.559270 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.558996 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 13:59:31.559270 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.558944 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 11 13:59:31.565881 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.565858 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2"] Sep 11 13:59:31.637267 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.637241 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 13:59:31.637680 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.637357 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" event={"ID":"6b7aadeb-5bf7-48b0-9169-7342cc8469ce","Type":"ContainerStarted","Data":"9590b94c45da0b9f9a398d6bcf526039b08c84f1735c809b219784b7eae1ccbe"} Sep 11 13:59:31.637680 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.637652 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:59:31.657183 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.657130 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" podStartSLOduration=40.161491026 podStartE2EDuration="42.657111017s" totalImagesPullingTime="2.495619991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:31.65617868 +0000 UTC m=+112.061159809" watchObservedRunningTime="2026-09-11 13:59:31.657111017 +0000 UTC m=+112.062092148" Sep 11 13:59:31.659402 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.659375 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-429m7\" (UniqueName: \"kubernetes.io/projected/c0038abf-1e13-4181-967c-a008f211149f-kube-api-access-429m7\") pod \"prometheus-operator-68bcd7f987-8cjb2\" (UID: \"c0038abf-1e13-4181-967c-a008f211149f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" Sep 11 13:59:31.659540 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.659421 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0038abf-1e13-4181-967c-a008f211149f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-8cjb2\" (UID: \"c0038abf-1e13-4181-967c-a008f211149f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" Sep 11 13:59:31.659540 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.659483 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0038abf-1e13-4181-967c-a008f211149f-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-8cjb2\" (UID: \"c0038abf-1e13-4181-967c-a008f211149f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" Sep 11 13:59:31.659540 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.659517 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0038abf-1e13-4181-967c-a008f211149f-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8cjb2\" (UID: \"c0038abf-1e13-4181-967c-a008f211149f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" Sep 11 13:59:31.701236 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.701208 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-qvnzs" Sep 11 13:59:31.760371 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.760336 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-429m7\" (UniqueName: \"kubernetes.io/projected/c0038abf-1e13-4181-967c-a008f211149f-kube-api-access-429m7\") pod \"prometheus-operator-68bcd7f987-8cjb2\" (UID: \"c0038abf-1e13-4181-967c-a008f211149f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" Sep 11 13:59:31.760569 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.760381 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0038abf-1e13-4181-967c-a008f211149f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-8cjb2\" (UID: \"c0038abf-1e13-4181-967c-a008f211149f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" Sep 11 13:59:31.760569 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.760491 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0038abf-1e13-4181-967c-a008f211149f-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-8cjb2\" (UID: \"c0038abf-1e13-4181-967c-a008f211149f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" Sep 11 13:59:31.760569 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.760555 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0038abf-1e13-4181-967c-a008f211149f-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8cjb2\" (UID: \"c0038abf-1e13-4181-967c-a008f211149f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" Sep 11 13:59:31.760966 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:31.760943 2566 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 11 13:59:31.761126 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:31.761012 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0038abf-1e13-4181-967c-a008f211149f-prometheus-operator-tls podName:c0038abf-1e13-4181-967c-a008f211149f nodeName:}" failed. No retries permitted until 2026-09-11 13:59:32.260993201 +0000 UTC m=+112.665974307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c0038abf-1e13-4181-967c-a008f211149f-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-8cjb2" (UID: "c0038abf-1e13-4181-967c-a008f211149f") : secret "prometheus-operator-tls" not found Sep 11 13:59:31.761203 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.761185 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0038abf-1e13-4181-967c-a008f211149f-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-8cjb2\" (UID: \"c0038abf-1e13-4181-967c-a008f211149f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" Sep 11 13:59:31.762943 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.762920 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0038abf-1e13-4181-967c-a008f211149f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-8cjb2\" (UID: \"c0038abf-1e13-4181-967c-a008f211149f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" Sep 11 13:59:31.770679 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.770653 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-429m7\" (UniqueName: \"kubernetes.io/projected/c0038abf-1e13-4181-967c-a008f211149f-kube-api-access-429m7\") pod \"prometheus-operator-68bcd7f987-8cjb2\" (UID: \"c0038abf-1e13-4181-967c-a008f211149f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" Sep 11 13:59:31.874941 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.874860 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-n29f8"] Sep 11 13:59:31.877126 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.877111 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-n29f8" Sep 11 13:59:31.879852 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.879829 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 13:59:31.880140 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.880127 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 13:59:31.880140 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.880136 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-jkwr8\"" Sep 11 13:59:31.889509 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.889485 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-n29f8"] Sep 11 13:59:31.962815 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:31.962762 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twdl5\" (UniqueName: \"kubernetes.io/projected/406c7df7-aeda-48a9-8663-f87d01d2b033-kube-api-access-twdl5\") pod \"downloads-76f99584f4-n29f8\" (UID: \"406c7df7-aeda-48a9-8663-f87d01d2b033\") " pod="openshift-console/downloads-76f99584f4-n29f8" Sep 11 13:59:32.064075 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:32.064017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twdl5\" (UniqueName: \"kubernetes.io/projected/406c7df7-aeda-48a9-8663-f87d01d2b033-kube-api-access-twdl5\") pod \"downloads-76f99584f4-n29f8\" (UID: \"406c7df7-aeda-48a9-8663-f87d01d2b033\") " pod="openshift-console/downloads-76f99584f4-n29f8" Sep 11 13:59:32.073499 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:32.073461 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twdl5\" (UniqueName: \"kubernetes.io/projected/406c7df7-aeda-48a9-8663-f87d01d2b033-kube-api-access-twdl5\") pod \"downloads-76f99584f4-n29f8\" (UID: \"406c7df7-aeda-48a9-8663-f87d01d2b033\") " pod="openshift-console/downloads-76f99584f4-n29f8" Sep 11 13:59:32.186036 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:32.185946 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-n29f8" Sep 11 13:59:32.265648 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:32.265601 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0038abf-1e13-4181-967c-a008f211149f-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8cjb2\" (UID: \"c0038abf-1e13-4181-967c-a008f211149f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" Sep 11 13:59:32.268116 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:32.268074 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0038abf-1e13-4181-967c-a008f211149f-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-8cjb2\" (UID: \"c0038abf-1e13-4181-967c-a008f211149f\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" Sep 11 13:59:32.464773 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:32.464691 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" Sep 11 13:59:32.485962 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:32.485937 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-n29f8"] Sep 11 13:59:32.488319 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:59:32.488296 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406c7df7_aeda_48a9_8663_f87d01d2b033.slice/crio-b6a65dcc8ac53f3356864951c379fc9231d21f0fe1cf24641030bc993a960ab5 WatchSource:0}: Error finding container b6a65dcc8ac53f3356864951c379fc9231d21f0fe1cf24641030bc993a960ab5: Status 404 returned error can't find the container with id b6a65dcc8ac53f3356864951c379fc9231d21f0fe1cf24641030bc993a960ab5 Sep 11 13:59:32.642153 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:32.642113 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-n29f8" event={"ID":"406c7df7-aeda-48a9-8663-f87d01d2b033","Type":"ContainerStarted","Data":"b6a65dcc8ac53f3356864951c379fc9231d21f0fe1cf24641030bc993a960ab5"} Sep 11 13:59:32.740692 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:32.740565 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2"] Sep 11 13:59:32.743035 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:59:32.743009 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0038abf_1e13_4181_967c_a008f211149f.slice/crio-ef0c4c9c0133b67eb7943cf67c4edc98d76f1cc78a7008ddfe97fcb127b731ce WatchSource:0}: Error finding container ef0c4c9c0133b67eb7943cf67c4edc98d76f1cc78a7008ddfe97fcb127b731ce: Status 404 returned error can't find the container with id ef0c4c9c0133b67eb7943cf67c4edc98d76f1cc78a7008ddfe97fcb127b731ce Sep 11 13:59:33.649423 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:33.649379 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" event={"ID":"c0038abf-1e13-4181-967c-a008f211149f","Type":"ContainerStarted","Data":"ef0c4c9c0133b67eb7943cf67c4edc98d76f1cc78a7008ddfe97fcb127b731ce"} Sep 11 13:59:34.655545 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:34.655502 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" event={"ID":"c0038abf-1e13-4181-967c-a008f211149f","Type":"ContainerStarted","Data":"8dcfc0c434aafffb06eec772fb60969874596a20a31bf61e801404f3be3ee87d"} Sep 11 13:59:34.655971 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:34.655568 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" event={"ID":"c0038abf-1e13-4181-967c-a008f211149f","Type":"ContainerStarted","Data":"c47c2e4da4e9c0ea4a6926f8c176aab8d428d377f3c6f63d0f448e87e9ab2bab"} Sep 11 13:59:34.674431 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:34.674373 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-8cjb2" podStartSLOduration=2.41122015 podStartE2EDuration="3.674356528s" totalImagesPullingTime="1.263136378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:34.673266963 +0000 UTC m=+115.078248091" watchObservedRunningTime="2026-09-11 13:59:34.674356528 +0000 UTC m=+115.079337656" Sep 11 13:59:37.270643 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.270608 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8qmlv"] Sep 11 13:59:37.273054 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.273028 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.276320 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.276296 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 13:59:37.276561 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.276534 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 13:59:37.276748 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.276536 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 13:59:37.277244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.277227 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wckx2\"" Sep 11 13:59:37.412287 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.412249 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-sys\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.412287 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.412302 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-wtmp\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.412551 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.412339 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.412551 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.412412 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-tls\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.412551 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.412448 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgxzc\" (UniqueName: \"kubernetes.io/projected/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-kube-api-access-xgxzc\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.412551 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.412474 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-metrics-client-ca\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.412551 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.412497 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.412551 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.412524 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-root\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.412779 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.412615 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-textfile\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.513627 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.513586 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.513627 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.513629 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-tls\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.513840 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.513652 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgxzc\" (UniqueName: \"kubernetes.io/projected/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-kube-api-access-xgxzc\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.513840 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.513675 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-metrics-client-ca\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.513840 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.513698 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.513840 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.513723 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-root\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.513840 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:37.513766 2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 13:59:37.513840 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.513785 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-textfile\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.513840 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:37.513836 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-tls podName:09f40fed-0af8-4fc9-8b29-fcb98e52e8c6 nodeName:}" failed. No retries permitted until 2026-09-11 13:59:38.013814377 +0000 UTC m=+118.418795498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-tls") pod "node-exporter-8qmlv" (UID: "09f40fed-0af8-4fc9-8b29-fcb98e52e8c6") : secret "node-exporter-tls" not found Sep 11 13:59:37.514250 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.513860 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-sys\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.514250 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.513904 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-wtmp\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.514250 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.514067 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-wtmp\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.514250 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.514133 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-sys\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.514250 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.514139 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-textfile\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.514250 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.514175 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-root\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.514646 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.514624 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-metrics-client-ca\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.514856 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.514830 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.516395 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.516373 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:37.523884 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:37.523832 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgxzc\" (UniqueName: \"kubernetes.io/projected/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-kube-api-access-xgxzc\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:38.017904 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:38.017867 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-tls\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:38.020669 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:38.020643 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09f40fed-0af8-4fc9-8b29-fcb98e52e8c6-node-exporter-tls\") pod \"node-exporter-8qmlv\" (UID: \"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6\") " pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:38.184564 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:38.184528 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8qmlv" Sep 11 13:59:38.196479 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:59:38.196438 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f40fed_0af8_4fc9_8b29_fcb98e52e8c6.slice/crio-0f1afe0f4f0ca6af63b12621a202b5587a1b7c72b0009469743e5e9335cc8d81 WatchSource:0}: Error finding container 0f1afe0f4f0ca6af63b12621a202b5587a1b7c72b0009469743e5e9335cc8d81: Status 404 returned error can't find the container with id 0f1afe0f4f0ca6af63b12621a202b5587a1b7c72b0009469743e5e9335cc8d81 Sep 11 13:59:38.672685 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:38.672642 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qmlv" event={"ID":"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6","Type":"ContainerStarted","Data":"0f1afe0f4f0ca6af63b12621a202b5587a1b7c72b0009469743e5e9335cc8d81"} Sep 11 13:59:39.678674 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:39.678577 2566 generic.go:356] "Generic (PLEG): container finished" podID="09f40fed-0af8-4fc9-8b29-fcb98e52e8c6" containerID="b0d7517a7efbe17051df3146b53a96dbaa58e6f4ac27132903c176c74b2ae9b8" exitCode=0 Sep 11 13:59:39.679109 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:39.678684 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qmlv" event={"ID":"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6","Type":"ContainerDied","Data":"b0d7517a7efbe17051df3146b53a96dbaa58e6f4ac27132903c176c74b2ae9b8"} Sep 11 13:59:40.042717 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.042688 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5ffffdd878-wn7tn"] Sep 11 13:59:40.046152 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.046128 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.049510 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.049485 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-c5jpz\"" Sep 11 13:59:40.049943 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.049921 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 11 13:59:40.050454 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.050428 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-37gd38r3pu00g\"" Sep 11 13:59:40.050560 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.050463 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 11 13:59:40.050560 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.050524 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 11 13:59:40.050678 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.050524 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 11 13:59:40.050741 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.050727 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 11 13:59:40.061569 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.061546 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5ffffdd878-wn7tn"] Sep 11 13:59:40.136210 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.136178 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.136413 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.136225 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-grpc-tls\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.136413 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.136256 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.136413 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.136284 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-metrics-client-ca\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.136413 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.136311 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-thanos-querier-tls\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.136591 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.136429 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.136591 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.136462 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.136591 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.136480 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp8sd\" (UniqueName: \"kubernetes.io/projected/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-kube-api-access-bp8sd\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.237379 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.237336 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.237550 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.237394 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.237550 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.237430 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8sd\" (UniqueName: \"kubernetes.io/projected/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-kube-api-access-bp8sd\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.237550 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.237466 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.239365 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.237840 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-grpc-tls\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.239365 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.237888 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.239365 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.237917 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-metrics-client-ca\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.239365 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.237946 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-thanos-querier-tls\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.239365 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.239305 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-metrics-client-ca\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.240575 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.240552 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.241112 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.241074 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-grpc-tls\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.241200 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.241076 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.241200 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.241127 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-thanos-querier-tls\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.241200 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.241127 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.241489 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.241466 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.247598 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.247579 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8sd\" (UniqueName: \"kubernetes.io/projected/dd1efa0e-bbe9-41a1-9b22-16cc926202a5-kube-api-access-bp8sd\") pod \"thanos-querier-5ffffdd878-wn7tn\" (UID: \"dd1efa0e-bbe9-41a1-9b22-16cc926202a5\") " pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.358419 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.358339 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:40.653060 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.652981 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5ffffdd878-wn7tn"] Sep 11 13:59:40.658244 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:59:40.658218 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1efa0e_bbe9_41a1_9b22_16cc926202a5.slice/crio-d681492f7411b889221d883f3194ac43312314e5ba73eeab817667cfdb314de8 WatchSource:0}: Error finding container d681492f7411b889221d883f3194ac43312314e5ba73eeab817667cfdb314de8: Status 404 returned error can't find the container with id d681492f7411b889221d883f3194ac43312314e5ba73eeab817667cfdb314de8 Sep 11 13:59:40.686006 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.685976 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qmlv" event={"ID":"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6","Type":"ContainerStarted","Data":"3127f33e2d9f3aaf4e5668bf78551f685a37fad10544a122e39786796d8bba9f"} Sep 11 13:59:40.686431 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.686038 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8qmlv" event={"ID":"09f40fed-0af8-4fc9-8b29-fcb98e52e8c6","Type":"ContainerStarted","Data":"a9fb8baa81f72c4505904c3ceaa9e23ab94072ae1456db962470506e5a0529e7"} Sep 11 13:59:40.687361 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:40.687330 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" event={"ID":"dd1efa0e-bbe9-41a1-9b22-16cc926202a5","Type":"ContainerStarted","Data":"d681492f7411b889221d883f3194ac43312314e5ba73eeab817667cfdb314de8"} Sep 11 13:59:41.771886 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:41.771387 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8qmlv" podStartSLOduration=3.6373798280000003 podStartE2EDuration="4.771361691s" totalImagesPullingTime="1.053924345s" totalInitContainerRuntime="80.057518ms" isStatefulPod=true podCreationTimestamp="2026-09-11 13:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:40.716391088 +0000 UTC m=+121.121372216" watchObservedRunningTime="2026-09-11 13:59:41.771361691 +0000 UTC m=+122.176342820" Sep 11 13:59:41.772379 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:41.772158 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-989k8"] Sep 11 13:59:41.775270 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:41.775246 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-989k8" Sep 11 13:59:41.779532 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:41.779508 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 11 13:59:41.779750 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:41.779733 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-r4md4\"" Sep 11 13:59:41.787744 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:41.787717 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-989k8"] Sep 11 13:59:41.952584 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:41.952550 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwbmv\" (UniqueName: \"kubernetes.io/projected/e83c0e3b-9a72-4b5b-af23-5a5f61680944-kube-api-access-lwbmv\") pod \"monitoring-plugin-5d85778b9d-989k8\" (UID: \"e83c0e3b-9a72-4b5b-af23-5a5f61680944\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-989k8" Sep 11 13:59:41.952584 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:41.952590 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e83c0e3b-9a72-4b5b-af23-5a5f61680944-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-989k8\" (UID: \"e83c0e3b-9a72-4b5b-af23-5a5f61680944\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-989k8" Sep 11 13:59:42.054033 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:42.053939 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwbmv\" (UniqueName: \"kubernetes.io/projected/e83c0e3b-9a72-4b5b-af23-5a5f61680944-kube-api-access-lwbmv\") pod \"monitoring-plugin-5d85778b9d-989k8\" (UID: \"e83c0e3b-9a72-4b5b-af23-5a5f61680944\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-989k8" Sep 11 13:59:42.054247 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:42.054036 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e83c0e3b-9a72-4b5b-af23-5a5f61680944-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-989k8\" (UID: \"e83c0e3b-9a72-4b5b-af23-5a5f61680944\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-989k8" Sep 11 13:59:42.057429 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:42.057390 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e83c0e3b-9a72-4b5b-af23-5a5f61680944-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-989k8\" (UID: \"e83c0e3b-9a72-4b5b-af23-5a5f61680944\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-989k8" Sep 11 13:59:42.064590 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:42.063605 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwbmv\" (UniqueName: \"kubernetes.io/projected/e83c0e3b-9a72-4b5b-af23-5a5f61680944-kube-api-access-lwbmv\") pod \"monitoring-plugin-5d85778b9d-989k8\" (UID: \"e83c0e3b-9a72-4b5b-af23-5a5f61680944\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-989k8" Sep 11 13:59:42.091564 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:42.091529 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-989k8" Sep 11 13:59:47.983234 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:47.983206 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:59:48.814220 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:48.814178 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs\") pod \"network-metrics-daemon-hphpv\" (UID: \"98e24790-f943-4aa7-b079-b8940fb02a82\") " pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:59:48.817007 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:48.816976 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98e24790-f943-4aa7-b079-b8940fb02a82-metrics-certs\") pod \"network-metrics-daemon-hphpv\" (UID: \"98e24790-f943-4aa7-b079-b8940fb02a82\") " pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:59:49.110513 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.110432 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qh858\"" Sep 11 13:59:49.118665 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.118644 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hphpv" Sep 11 13:59:49.587465 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.587425 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b5d8667b6-zs7tw"] Sep 11 13:59:49.590557 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.590533 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.594079 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.594054 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 13:59:49.594244 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.594121 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 13:59:49.594395 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.594374 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-242xz\"" Sep 11 13:59:49.594510 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.594477 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 13:59:49.594585 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.594568 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 13:59:49.596357 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.596337 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 13:59:49.602591 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.602477 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 13:59:49.615521 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.615390 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5d8667b6-zs7tw"] Sep 11 13:59:49.621117 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.621070 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-serving-cert\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.621289 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.621271 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-config\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.621404 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.621390 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-service-ca\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.621536 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.621516 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-trusted-ca-bundle\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.621621 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.621570 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-oauth-config\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.621621 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.621605 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbj8x\" (UniqueName: \"kubernetes.io/projected/de2806c8-2a3d-4c22-b6ec-de23fe511977-kube-api-access-pbj8x\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.621731 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.621648 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-oauth-serving-cert\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.722061 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.721998 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-serving-cert\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.722061 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.722037 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-config\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.722627 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.722064 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-service-ca\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.722627 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.722119 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-trusted-ca-bundle\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.722627 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.722161 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-oauth-config\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.722627 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.722197 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbj8x\" (UniqueName: \"kubernetes.io/projected/de2806c8-2a3d-4c22-b6ec-de23fe511977-kube-api-access-pbj8x\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.722627 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.722242 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-oauth-serving-cert\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.724538 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.723411 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-trusted-ca-bundle\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.724538 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.723440 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-config\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.724538 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.723818 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-oauth-serving-cert\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.724538 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.724211 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-service-ca\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.725408 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.725263 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-serving-cert\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.726021 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.726001 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-oauth-config\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.726143 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.726118 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" event={"ID":"dd1efa0e-bbe9-41a1-9b22-16cc926202a5","Type":"ContainerStarted","Data":"b146e45fd75982de5b9146b7b969452ba9113c9f2fc09c85abd0fd973f6eef15"} Sep 11 13:59:49.732723 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.732686 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbj8x\" (UniqueName: \"kubernetes.io/projected/de2806c8-2a3d-4c22-b6ec-de23fe511977-kube-api-access-pbj8x\") pod \"console-b5d8667b6-zs7tw\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:49.877237 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.877026 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-989k8"] Sep 11 13:59:49.881314 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.880344 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hphpv"] Sep 11 13:59:49.881314 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:59:49.880551 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83c0e3b_9a72_4b5b_af23_5a5f61680944.slice/crio-5fd5da26f2c9431abb013d442cd4b25684940a93b06b7f6baaa11b8121564d06 WatchSource:0}: Error finding container 5fd5da26f2c9431abb013d442cd4b25684940a93b06b7f6baaa11b8121564d06: Status 404 returned error can't find the container with id 5fd5da26f2c9431abb013d442cd4b25684940a93b06b7f6baaa11b8121564d06 Sep 11 13:59:49.882993 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:59:49.882881 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e24790_f943_4aa7_b079_b8940fb02a82.slice/crio-ecfa068ccb211a8b2c94ac53ef8578e2c5ce8de704f392904be81a31f5a14457 WatchSource:0}: Error finding container ecfa068ccb211a8b2c94ac53ef8578e2c5ce8de704f392904be81a31f5a14457: Status 404 returned error can't find the container with id ecfa068ccb211a8b2c94ac53ef8578e2c5ce8de704f392904be81a31f5a14457 Sep 11 13:59:49.924427 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:49.924211 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:50.224468 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:50.224440 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5d8667b6-zs7tw"] Sep 11 13:59:50.227837 ip-10-0-136-92 kubenswrapper[2566]: W0911 13:59:50.227793 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2806c8_2a3d_4c22_b6ec_de23fe511977.slice/crio-5433240ed8722cb557b1793bc41efa01b0eeabadc652449a0ba369ae0a1d086d WatchSource:0}: Error finding container 5433240ed8722cb557b1793bc41efa01b0eeabadc652449a0ba369ae0a1d086d: Status 404 returned error can't find the container with id 5433240ed8722cb557b1793bc41efa01b0eeabadc652449a0ba369ae0a1d086d Sep 11 13:59:50.737466 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:50.736988 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-989k8" event={"ID":"e83c0e3b-9a72-4b5b-af23-5a5f61680944","Type":"ContainerStarted","Data":"5fd5da26f2c9431abb013d442cd4b25684940a93b06b7f6baaa11b8121564d06"} Sep 11 13:59:50.742468 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:50.741952 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hphpv" event={"ID":"98e24790-f943-4aa7-b079-b8940fb02a82","Type":"ContainerStarted","Data":"ecfa068ccb211a8b2c94ac53ef8578e2c5ce8de704f392904be81a31f5a14457"} Sep 11 13:59:50.747378 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:50.747314 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5d8667b6-zs7tw" event={"ID":"de2806c8-2a3d-4c22-b6ec-de23fe511977","Type":"ContainerStarted","Data":"5433240ed8722cb557b1793bc41efa01b0eeabadc652449a0ba369ae0a1d086d"} Sep 11 13:59:50.749982 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:50.749859 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-n29f8" event={"ID":"406c7df7-aeda-48a9-8663-f87d01d2b033","Type":"ContainerStarted","Data":"ae07f337a964c97be266e93278226bc511402f1e013830c541b0932746fd9644"} Sep 11 13:59:50.752692 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:50.752668 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-n29f8" Sep 11 13:59:50.756725 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:50.756611 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" event={"ID":"dd1efa0e-bbe9-41a1-9b22-16cc926202a5","Type":"ContainerStarted","Data":"7c1775313d779e75a7abe843fd7bb9a70e3990af1698b115d6dfe524d5d688ad"} Sep 11 13:59:50.756725 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:50.756679 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" event={"ID":"dd1efa0e-bbe9-41a1-9b22-16cc926202a5","Type":"ContainerStarted","Data":"1633a1d83abd7094241b4a505e2ed988dac7f14ac0f552dd8ad22117c6b651da"} Sep 11 13:59:50.761299 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:50.761229 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-n29f8" Sep 11 13:59:50.780295 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:50.779601 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-n29f8" podStartSLOduration=2.601961204 podStartE2EDuration="19.779580983s" totalImagesPullingTime="17.177619779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:50.7766208 +0000 UTC m=+131.181601925" watchObservedRunningTime="2026-09-11 13:59:50.779580983 +0000 UTC m=+131.184562111" Sep 11 13:59:52.770823 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:52.770741 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" event={"ID":"dd1efa0e-bbe9-41a1-9b22-16cc926202a5","Type":"ContainerStarted","Data":"2cd8577941d66d8b31d1df5ec9d42e9b3f63a65bbc6d57468334db9fdf1c5c8a"} Sep 11 13:59:52.771308 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:52.770848 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" event={"ID":"dd1efa0e-bbe9-41a1-9b22-16cc926202a5","Type":"ContainerStarted","Data":"586547bf6f068de0e8e937cd0a51b3ad4b091b31c8e1295deb05bbe7b94160dc"} Sep 11 13:59:52.774405 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:52.773003 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-989k8" event={"ID":"e83c0e3b-9a72-4b5b-af23-5a5f61680944","Type":"ContainerStarted","Data":"ef9b6296fd60e1d5e68671c925269fdc9f6967ef696b2b84e94e3bfd7ff9387f"} Sep 11 13:59:52.775961 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:52.775943 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-989k8" Sep 11 13:59:52.778938 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:52.778914 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hphpv" event={"ID":"98e24790-f943-4aa7-b079-b8940fb02a82","Type":"ContainerStarted","Data":"019007520f2b3951f178efdafc92d15d602720dd54870c8a6a4b101030ad4ae2"} Sep 11 13:59:52.784458 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:52.784441 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-989k8" Sep 11 13:59:52.803146 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:52.803026 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-989k8" podStartSLOduration=9.18604237 podStartE2EDuration="11.80300458s" totalImagesPullingTime="2.61696221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:52.8009901 +0000 UTC m=+133.205971232" watchObservedRunningTime="2026-09-11 13:59:52.80300458 +0000 UTC m=+133.207985709" Sep 11 13:59:52.995446 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:52.995344 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" podUID="d70e5b7a-dfee-4881-a8fb-24f858af429e" containerName="registry" containerID="cri-o://520b6d66fa5bf03e43555274337f16a8bf6bea92aecdb5376c246c61ba7f28e8" gracePeriod=30 Sep 11 13:59:54.318987 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.318948 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:59:54.363343 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.363267 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-certificates\") pod \"d70e5b7a-dfee-4881-a8fb-24f858af429e\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " Sep 11 13:59:54.363343 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.363322 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d70e5b7a-dfee-4881-a8fb-24f858af429e-trusted-ca\") pod \"d70e5b7a-dfee-4881-a8fb-24f858af429e\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " Sep 11 13:59:54.363547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.363361 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d70e5b7a-dfee-4881-a8fb-24f858af429e-installation-pull-secrets\") pod \"d70e5b7a-dfee-4881-a8fb-24f858af429e\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " Sep 11 13:59:54.363547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.363438 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-bound-sa-token\") pod \"d70e5b7a-dfee-4881-a8fb-24f858af429e\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " Sep 11 13:59:54.363547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.363474 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d70e5b7a-dfee-4881-a8fb-24f858af429e-ca-trust-extracted\") pod \"d70e5b7a-dfee-4881-a8fb-24f858af429e\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " Sep 11 13:59:54.363547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.363503 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls\") pod \"d70e5b7a-dfee-4881-a8fb-24f858af429e\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " Sep 11 13:59:54.363547 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.363531 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-742js\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-kube-api-access-742js\") pod \"d70e5b7a-dfee-4881-a8fb-24f858af429e\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " Sep 11 13:59:54.363758 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.363569 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d70e5b7a-dfee-4881-a8fb-24f858af429e-image-registry-private-configuration\") pod \"d70e5b7a-dfee-4881-a8fb-24f858af429e\" (UID: \"d70e5b7a-dfee-4881-a8fb-24f858af429e\") " Sep 11 13:59:54.363758 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.363614 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d70e5b7a-dfee-4881-a8fb-24f858af429e" (UID: "d70e5b7a-dfee-4881-a8fb-24f858af429e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:59:54.363865 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.363802 2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-certificates\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 13:59:54.364611 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.364582 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70e5b7a-dfee-4881-a8fb-24f858af429e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d70e5b7a-dfee-4881-a8fb-24f858af429e" (UID: "d70e5b7a-dfee-4881-a8fb-24f858af429e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:59:54.367364 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.367334 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-kube-api-access-742js" (OuterVolumeSpecName: "kube-api-access-742js") pod "d70e5b7a-dfee-4881-a8fb-24f858af429e" (UID: "d70e5b7a-dfee-4881-a8fb-24f858af429e"). InnerVolumeSpecName "kube-api-access-742js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:59:54.367473 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.367408 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70e5b7a-dfee-4881-a8fb-24f858af429e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d70e5b7a-dfee-4881-a8fb-24f858af429e" (UID: "d70e5b7a-dfee-4881-a8fb-24f858af429e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:59:54.369130 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.369074 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d70e5b7a-dfee-4881-a8fb-24f858af429e" (UID: "d70e5b7a-dfee-4881-a8fb-24f858af429e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:59:54.369130 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.369119 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d70e5b7a-dfee-4881-a8fb-24f858af429e" (UID: "d70e5b7a-dfee-4881-a8fb-24f858af429e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:59:54.370730 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.370706 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70e5b7a-dfee-4881-a8fb-24f858af429e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d70e5b7a-dfee-4881-a8fb-24f858af429e" (UID: "d70e5b7a-dfee-4881-a8fb-24f858af429e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:59:54.374505 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.374482 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70e5b7a-dfee-4881-a8fb-24f858af429e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d70e5b7a-dfee-4881-a8fb-24f858af429e" (UID: "d70e5b7a-dfee-4881-a8fb-24f858af429e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 13:59:54.465253 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.465216 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d70e5b7a-dfee-4881-a8fb-24f858af429e-trusted-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 13:59:54.465253 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.465249 2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d70e5b7a-dfee-4881-a8fb-24f858af429e-installation-pull-secrets\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 13:59:54.465475 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.465263 2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-bound-sa-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 13:59:54.465475 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.465277 2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d70e5b7a-dfee-4881-a8fb-24f858af429e-ca-trust-extracted\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 13:59:54.465475 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.465288 2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-registry-tls\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 13:59:54.465475 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.465299 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-742js\" (UniqueName: \"kubernetes.io/projected/d70e5b7a-dfee-4881-a8fb-24f858af429e-kube-api-access-742js\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 13:59:54.465475 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.465310 2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d70e5b7a-dfee-4881-a8fb-24f858af429e-image-registry-private-configuration\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 13:59:54.791861 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.791817 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" event={"ID":"dd1efa0e-bbe9-41a1-9b22-16cc926202a5","Type":"ContainerStarted","Data":"7f5ca1516b4b23e176d34ec6cf9606af90a0117bc11c0ac5b22789bb3fd04636"} Sep 11 13:59:54.792127 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.792043 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:54.793453 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.793423 2566 generic.go:356] "Generic (PLEG): container finished" podID="d70e5b7a-dfee-4881-a8fb-24f858af429e" containerID="520b6d66fa5bf03e43555274337f16a8bf6bea92aecdb5376c246c61ba7f28e8" exitCode=0 Sep 11 13:59:54.793585 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.793473 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" Sep 11 13:59:54.793585 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.793513 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" event={"ID":"d70e5b7a-dfee-4881-a8fb-24f858af429e","Type":"ContainerDied","Data":"520b6d66fa5bf03e43555274337f16a8bf6bea92aecdb5376c246c61ba7f28e8"} Sep 11 13:59:54.793585 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.793567 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59c887d6c5-wm7zl" event={"ID":"d70e5b7a-dfee-4881-a8fb-24f858af429e","Type":"ContainerDied","Data":"1c48e54900b5b1cbb7de2eb56cdff7f4c32a769a716895d791d9243e4fc0b8df"} Sep 11 13:59:54.793738 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.793598 2566 scope.go:117] "RemoveContainer" containerID="520b6d66fa5bf03e43555274337f16a8bf6bea92aecdb5376c246c61ba7f28e8" Sep 11 13:59:54.796152 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.796119 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hphpv" event={"ID":"98e24790-f943-4aa7-b079-b8940fb02a82","Type":"ContainerStarted","Data":"caf4fd29d4de56fd6b35cc458f86329a361e06221a4d114bf7c9e714c951432c"} Sep 11 13:59:54.800148 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.800114 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5d8667b6-zs7tw" event={"ID":"de2806c8-2a3d-4c22-b6ec-de23fe511977","Type":"ContainerStarted","Data":"0366c385f659dc2281837d79b77a557ff4fc2646155530313a55690032d44482"} Sep 11 13:59:54.800597 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.800572 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" Sep 11 13:59:54.804716 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.804690 2566 scope.go:117] "RemoveContainer" containerID="520b6d66fa5bf03e43555274337f16a8bf6bea92aecdb5376c246c61ba7f28e8" Sep 11 13:59:54.805120 ip-10-0-136-92 kubenswrapper[2566]: E0911 13:59:54.805055 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520b6d66fa5bf03e43555274337f16a8bf6bea92aecdb5376c246c61ba7f28e8\": container with ID starting with 520b6d66fa5bf03e43555274337f16a8bf6bea92aecdb5376c246c61ba7f28e8 not found: ID does not exist" containerID="520b6d66fa5bf03e43555274337f16a8bf6bea92aecdb5376c246c61ba7f28e8" Sep 11 13:59:54.805213 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.805127 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520b6d66fa5bf03e43555274337f16a8bf6bea92aecdb5376c246c61ba7f28e8"} err="failed to get container status \"520b6d66fa5bf03e43555274337f16a8bf6bea92aecdb5376c246c61ba7f28e8\": rpc error: code = NotFound desc = could not find container \"520b6d66fa5bf03e43555274337f16a8bf6bea92aecdb5376c246c61ba7f28e8\": container with ID starting with 520b6d66fa5bf03e43555274337f16a8bf6bea92aecdb5376c246c61ba7f28e8 not found: ID does not exist" Sep 11 13:59:54.827225 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.827149 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5ffffdd878-wn7tn" podStartSLOduration=3.39595126 podStartE2EDuration="14.827127706s" totalImagesPullingTime="11.431176446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:59:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:54.822881188 +0000 UTC m=+135.227862308" watchObservedRunningTime="2026-09-11 13:59:54.827127706 +0000 UTC m=+135.232108837" Sep 11 13:59:54.888682 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.888623 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b5d8667b6-zs7tw" podStartSLOduration=1.5991191790000001 podStartE2EDuration="5.888605285s" totalImagesPullingTime="4.289486106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:54.870492095 +0000 UTC m=+135.275473225" watchObservedRunningTime="2026-09-11 13:59:54.888605285 +0000 UTC m=+135.293586413" Sep 11 13:59:54.888877 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.888799 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hphpv" podStartSLOduration=132.272222198 podStartE2EDuration="2m14.888793664s" totalImagesPullingTime="2.616571466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:54.886352488 +0000 UTC m=+135.291333616" watchObservedRunningTime="2026-09-11 13:59:54.888793664 +0000 UTC m=+135.293774792" Sep 11 13:59:54.902401 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.902369 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59c887d6c5-wm7zl"] Sep 11 13:59:54.906929 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:54.906893 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59c887d6c5-wm7zl"] Sep 11 13:59:56.097858 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:56.097820 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d70e5b7a-dfee-4881-a8fb-24f858af429e" path="/var/lib/kubelet/pods/d70e5b7a-dfee-4881-a8fb-24f858af429e/volumes" Sep 11 13:59:59.924400 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:59.924365 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:59.924813 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:59.924456 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 13:59:59.929030 ip-10-0-136-92 kubenswrapper[2566]: I0911 13:59:59.929007 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 14:00:00.832944 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:00:00.832900 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 14:00:08.861886 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:00:08.861850 2566 generic.go:356] "Generic (PLEG): container finished" podID="89a2ef95-796d-4a09-97be-fdaa2d981b30" containerID="5b46455c47655119b757787868ee567ec8e380216c5cfcd7001e980d60a70f5d" exitCode=0 Sep 11 14:00:08.862299 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:00:08.861908 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h9mh7" event={"ID":"89a2ef95-796d-4a09-97be-fdaa2d981b30","Type":"ContainerDied","Data":"5b46455c47655119b757787868ee567ec8e380216c5cfcd7001e980d60a70f5d"} Sep 11 14:00:08.862299 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:00:08.862211 2566 scope.go:117] "RemoveContainer" containerID="5b46455c47655119b757787868ee567ec8e380216c5cfcd7001e980d60a70f5d" Sep 11 14:00:09.867964 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:00:09.867923 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-h9mh7" event={"ID":"89a2ef95-796d-4a09-97be-fdaa2d981b30","Type":"ContainerStarted","Data":"8e554308066f9850228ce38e70e64873efb6c07b7ce4563308dd354230d27541"} Sep 11 14:01:15.280097 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:15.280057 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b5d8667b6-zs7tw"] Sep 11 14:01:40.299980 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.299917 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b5d8667b6-zs7tw" podUID="de2806c8-2a3d-4c22-b6ec-de23fe511977" containerName="console" containerID="cri-o://0366c385f659dc2281837d79b77a557ff4fc2646155530313a55690032d44482" gracePeriod=15 Sep 11 14:01:40.538551 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.538528 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 14:01:40.667049 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.666950 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-config\") pod \"de2806c8-2a3d-4c22-b6ec-de23fe511977\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " Sep 11 14:01:40.667049 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.666994 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-trusted-ca-bundle\") pod \"de2806c8-2a3d-4c22-b6ec-de23fe511977\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " Sep 11 14:01:40.667049 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.667017 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-oauth-config\") pod \"de2806c8-2a3d-4c22-b6ec-de23fe511977\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " Sep 11 14:01:40.667049 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.667041 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-oauth-serving-cert\") pod \"de2806c8-2a3d-4c22-b6ec-de23fe511977\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " Sep 11 14:01:40.667404 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.667170 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbj8x\" (UniqueName: \"kubernetes.io/projected/de2806c8-2a3d-4c22-b6ec-de23fe511977-kube-api-access-pbj8x\") pod \"de2806c8-2a3d-4c22-b6ec-de23fe511977\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " Sep 11 14:01:40.667404 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.667228 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-serving-cert\") pod \"de2806c8-2a3d-4c22-b6ec-de23fe511977\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " Sep 11 14:01:40.667404 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.667279 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-service-ca\") pod \"de2806c8-2a3d-4c22-b6ec-de23fe511977\" (UID: \"de2806c8-2a3d-4c22-b6ec-de23fe511977\") " Sep 11 14:01:40.667551 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.667448 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-config" (OuterVolumeSpecName: "console-config") pod "de2806c8-2a3d-4c22-b6ec-de23fe511977" (UID: "de2806c8-2a3d-4c22-b6ec-de23fe511977"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:01:40.667610 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.667574 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "de2806c8-2a3d-4c22-b6ec-de23fe511977" (UID: "de2806c8-2a3d-4c22-b6ec-de23fe511977"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:01:40.667694 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.667581 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "de2806c8-2a3d-4c22-b6ec-de23fe511977" (UID: "de2806c8-2a3d-4c22-b6ec-de23fe511977"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:01:40.667780 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.667758 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-service-ca" (OuterVolumeSpecName: "service-ca") pod "de2806c8-2a3d-4c22-b6ec-de23fe511977" (UID: "de2806c8-2a3d-4c22-b6ec-de23fe511977"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:01:40.669543 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.669516 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "de2806c8-2a3d-4c22-b6ec-de23fe511977" (UID: "de2806c8-2a3d-4c22-b6ec-de23fe511977"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:01:40.669667 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.669601 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "de2806c8-2a3d-4c22-b6ec-de23fe511977" (UID: "de2806c8-2a3d-4c22-b6ec-de23fe511977"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:01:40.669667 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.669604 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2806c8-2a3d-4c22-b6ec-de23fe511977-kube-api-access-pbj8x" (OuterVolumeSpecName: "kube-api-access-pbj8x") pod "de2806c8-2a3d-4c22-b6ec-de23fe511977" (UID: "de2806c8-2a3d-4c22-b6ec-de23fe511977"). InnerVolumeSpecName "kube-api-access-pbj8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:01:40.768567 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.768532 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:01:40.768567 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.768560 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-config\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:01:40.768567 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.768571 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-trusted-ca-bundle\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:01:40.768795 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.768581 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-oauth-config\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:01:40.768795 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.768590 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de2806c8-2a3d-4c22-b6ec-de23fe511977-oauth-serving-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:01:40.768795 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.768598 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbj8x\" (UniqueName: \"kubernetes.io/projected/de2806c8-2a3d-4c22-b6ec-de23fe511977-kube-api-access-pbj8x\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:01:40.768795 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:40.768608 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de2806c8-2a3d-4c22-b6ec-de23fe511977-console-serving-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:01:41.199293 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:41.199254 2566 generic.go:356] "Generic (PLEG): container finished" podID="de2806c8-2a3d-4c22-b6ec-de23fe511977" containerID="0366c385f659dc2281837d79b77a557ff4fc2646155530313a55690032d44482" exitCode=0 Sep 11 14:01:41.199624 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:41.199297 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5d8667b6-zs7tw" Sep 11 14:01:41.199624 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:41.199290 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5d8667b6-zs7tw" event={"ID":"de2806c8-2a3d-4c22-b6ec-de23fe511977","Type":"ContainerDied","Data":"0366c385f659dc2281837d79b77a557ff4fc2646155530313a55690032d44482"} Sep 11 14:01:41.199624 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:41.199410 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5d8667b6-zs7tw" event={"ID":"de2806c8-2a3d-4c22-b6ec-de23fe511977","Type":"ContainerDied","Data":"5433240ed8722cb557b1793bc41efa01b0eeabadc652449a0ba369ae0a1d086d"} Sep 11 14:01:41.199624 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:41.199434 2566 scope.go:117] "RemoveContainer" containerID="0366c385f659dc2281837d79b77a557ff4fc2646155530313a55690032d44482" Sep 11 14:01:41.208079 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:41.208064 2566 scope.go:117] "RemoveContainer" containerID="0366c385f659dc2281837d79b77a557ff4fc2646155530313a55690032d44482" Sep 11 14:01:41.208351 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:01:41.208335 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0366c385f659dc2281837d79b77a557ff4fc2646155530313a55690032d44482\": container with ID starting with 0366c385f659dc2281837d79b77a557ff4fc2646155530313a55690032d44482 not found: ID does not exist" containerID="0366c385f659dc2281837d79b77a557ff4fc2646155530313a55690032d44482" Sep 11 14:01:41.208395 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:41.208359 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0366c385f659dc2281837d79b77a557ff4fc2646155530313a55690032d44482"} err="failed to get container status \"0366c385f659dc2281837d79b77a557ff4fc2646155530313a55690032d44482\": rpc error: code = NotFound desc = could not find container \"0366c385f659dc2281837d79b77a557ff4fc2646155530313a55690032d44482\": container with ID starting with 0366c385f659dc2281837d79b77a557ff4fc2646155530313a55690032d44482 not found: ID does not exist" Sep 11 14:01:41.219671 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:41.219651 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b5d8667b6-zs7tw"] Sep 11 14:01:41.222907 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:41.222885 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b5d8667b6-zs7tw"] Sep 11 14:01:42.095932 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:01:42.095900 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de2806c8-2a3d-4c22-b6ec-de23fe511977" path="/var/lib/kubelet/pods/de2806c8-2a3d-4c22-b6ec-de23fe511977/volumes" Sep 11 14:02:16.448063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.447982 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c6c86cbd-j5stx"] Sep 11 14:02:16.448612 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.448403 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2806c8-2a3d-4c22-b6ec-de23fe511977" containerName="console" Sep 11 14:02:16.448612 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.448424 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2806c8-2a3d-4c22-b6ec-de23fe511977" containerName="console" Sep 11 14:02:16.448612 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.448447 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d70e5b7a-dfee-4881-a8fb-24f858af429e" containerName="registry" Sep 11 14:02:16.448612 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.448453 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d70e5b7a-dfee-4881-a8fb-24f858af429e" containerName="registry" Sep 11 14:02:16.448612 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.448514 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d70e5b7a-dfee-4881-a8fb-24f858af429e" containerName="registry" Sep 11 14:02:16.448612 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.448523 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2806c8-2a3d-4c22-b6ec-de23fe511977" containerName="console" Sep 11 14:02:16.450728 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.450704 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.453286 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.453237 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 14:02:16.453432 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.453324 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 14:02:16.454070 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.454048 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 14:02:16.454207 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.454079 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 14:02:16.454207 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.454119 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-242xz\"" Sep 11 14:02:16.454207 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.454194 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 14:02:16.459910 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.459891 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 14:02:16.467248 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.467219 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c6c86cbd-j5stx"] Sep 11 14:02:16.544009 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.543977 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-console-config\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.544194 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.544023 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-oauth-serving-cert\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.544194 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.544044 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f7fac22-3817-4831-a873-a1b09ef35696-console-oauth-config\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.544194 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.544112 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-service-ca\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.544194 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.544160 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-866s2\" (UniqueName: \"kubernetes.io/projected/1f7fac22-3817-4831-a873-a1b09ef35696-kube-api-access-866s2\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.544194 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.544180 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-trusted-ca-bundle\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.544194 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.544195 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f7fac22-3817-4831-a873-a1b09ef35696-console-serving-cert\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.645197 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.645156 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-oauth-serving-cert\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.645197 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.645200 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f7fac22-3817-4831-a873-a1b09ef35696-console-oauth-config\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.645408 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.645384 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-service-ca\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.645470 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.645455 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-866s2\" (UniqueName: \"kubernetes.io/projected/1f7fac22-3817-4831-a873-a1b09ef35696-kube-api-access-866s2\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.645515 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.645493 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-trusted-ca-bundle\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.645564 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.645519 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f7fac22-3817-4831-a873-a1b09ef35696-console-serving-cert\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.645617 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.645568 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-console-config\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.645933 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.645908 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-oauth-serving-cert\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.646049 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.646028 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-service-ca\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.646303 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.646280 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-console-config\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.646559 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.646289 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-trusted-ca-bundle\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.647806 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.647784 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f7fac22-3817-4831-a873-a1b09ef35696-console-oauth-config\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.648117 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.648101 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f7fac22-3817-4831-a873-a1b09ef35696-console-serving-cert\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.655097 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.655065 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-866s2\" (UniqueName: \"kubernetes.io/projected/1f7fac22-3817-4831-a873-a1b09ef35696-kube-api-access-866s2\") pod \"console-6c6c86cbd-j5stx\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:16.760979 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:16.760933 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:17.050855 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:17.050822 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c6c86cbd-j5stx"] Sep 11 14:02:17.053861 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:02:17.053834 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7fac22_3817_4831_a873_a1b09ef35696.slice/crio-668282fb09a34144c74e10714614bab1c2d3217694a7aacc742e0262658d33b2 WatchSource:0}: Error finding container 668282fb09a34144c74e10714614bab1c2d3217694a7aacc742e0262658d33b2: Status 404 returned error can't find the container with id 668282fb09a34144c74e10714614bab1c2d3217694a7aacc742e0262658d33b2 Sep 11 14:02:17.329195 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:17.329113 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6c86cbd-j5stx" event={"ID":"1f7fac22-3817-4831-a873-a1b09ef35696","Type":"ContainerStarted","Data":"246dbae5b19c11bc2b4d2260b0294322f4dbb176d5df81afa53047e7bb9591fe"} Sep 11 14:02:17.329195 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:17.329161 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6c86cbd-j5stx" event={"ID":"1f7fac22-3817-4831-a873-a1b09ef35696","Type":"ContainerStarted","Data":"668282fb09a34144c74e10714614bab1c2d3217694a7aacc742e0262658d33b2"} Sep 11 14:02:17.354107 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:17.354042 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c6c86cbd-j5stx" podStartSLOduration=1.354021638 podStartE2EDuration="1.354021638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:02:17.353300434 +0000 UTC m=+277.758281562" watchObservedRunningTime="2026-09-11 14:02:17.354021638 +0000 UTC m=+277.759002765" Sep 11 14:02:26.761185 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:26.761150 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:26.761659 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:26.761205 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:26.765890 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:26.765868 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:27.368695 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:27.368667 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:02:40.016650 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:40.016622 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:02:40.027368 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:40.027324 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:02:40.033320 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:40.033293 2566 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 14:02:48.275906 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.275874 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8d9d854b-ncjls"] Sep 11 14:02:48.277865 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.277846 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8d9d854b-ncjls" Sep 11 14:02:48.284433 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.284411 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 11 14:02:48.284606 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.284463 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 14:02:48.284606 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.284411 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 14:02:48.285037 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.285019 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 14:02:48.304024 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.303993 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8d9d854b-ncjls"] Sep 11 14:02:48.383410 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.383375 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6fc19ea0-312e-4e36-a5bb-5f3103cde648-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8d9d854b-ncjls\" (UID: \"6fc19ea0-312e-4e36-a5bb-5f3103cde648\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8d9d854b-ncjls" Sep 11 14:02:48.383596 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.383446 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpjp4\" (UniqueName: \"kubernetes.io/projected/6fc19ea0-312e-4e36-a5bb-5f3103cde648-kube-api-access-zpjp4\") pod \"managed-serviceaccount-addon-agent-8d9d854b-ncjls\" (UID: \"6fc19ea0-312e-4e36-a5bb-5f3103cde648\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8d9d854b-ncjls" Sep 11 14:02:48.484820 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.484781 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpjp4\" (UniqueName: \"kubernetes.io/projected/6fc19ea0-312e-4e36-a5bb-5f3103cde648-kube-api-access-zpjp4\") pod \"managed-serviceaccount-addon-agent-8d9d854b-ncjls\" (UID: \"6fc19ea0-312e-4e36-a5bb-5f3103cde648\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8d9d854b-ncjls" Sep 11 14:02:48.484991 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.484830 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6fc19ea0-312e-4e36-a5bb-5f3103cde648-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8d9d854b-ncjls\" (UID: \"6fc19ea0-312e-4e36-a5bb-5f3103cde648\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8d9d854b-ncjls" Sep 11 14:02:48.487538 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.487519 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6fc19ea0-312e-4e36-a5bb-5f3103cde648-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8d9d854b-ncjls\" (UID: \"6fc19ea0-312e-4e36-a5bb-5f3103cde648\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8d9d854b-ncjls" Sep 11 14:02:48.507795 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.507763 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpjp4\" (UniqueName: \"kubernetes.io/projected/6fc19ea0-312e-4e36-a5bb-5f3103cde648-kube-api-access-zpjp4\") pod \"managed-serviceaccount-addon-agent-8d9d854b-ncjls\" (UID: \"6fc19ea0-312e-4e36-a5bb-5f3103cde648\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8d9d854b-ncjls" Sep 11 14:02:48.596839 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.596737 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8d9d854b-ncjls" Sep 11 14:02:48.870644 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.870620 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8d9d854b-ncjls"] Sep 11 14:02:48.872810 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:02:48.872768 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fc19ea0_312e_4e36_a5bb_5f3103cde648.slice/crio-14bf6ec5740604a3a62fb1efad619b658ed8daa9818ec01f0de691e27a2254fe WatchSource:0}: Error finding container 14bf6ec5740604a3a62fb1efad619b658ed8daa9818ec01f0de691e27a2254fe: Status 404 returned error can't find the container with id 14bf6ec5740604a3a62fb1efad619b658ed8daa9818ec01f0de691e27a2254fe Sep 11 14:02:48.874650 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:48.874632 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:02:49.451750 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:49.451716 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8d9d854b-ncjls" event={"ID":"6fc19ea0-312e-4e36-a5bb-5f3103cde648","Type":"ContainerStarted","Data":"14bf6ec5740604a3a62fb1efad619b658ed8daa9818ec01f0de691e27a2254fe"} Sep 11 14:02:52.464975 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:52.464939 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8d9d854b-ncjls" event={"ID":"6fc19ea0-312e-4e36-a5bb-5f3103cde648","Type":"ContainerStarted","Data":"8a622cf95e6f1e649be48e1c3366d62d78e343dab2279e97185a9d627c4c78a6"} Sep 11 14:02:52.483438 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:02:52.483379 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8d9d854b-ncjls" podStartSLOduration=1.54915913 podStartE2EDuration="4.483363558s" totalImagesPullingTime="2.934204428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:02:52.48145725 +0000 UTC m=+312.886438390" watchObservedRunningTime="2026-09-11 14:02:52.483363558 +0000 UTC m=+312.888344688" Sep 11 14:04:37.065418 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.065380 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn"] Sep 11 14:04:37.069492 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.069474 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" Sep 11 14:04:37.071980 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.071958 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:04:37.072816 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.072801 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-w8ql7\"" Sep 11 14:04:37.072907 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.072806 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:04:37.096601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.096576 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn"] Sep 11 14:04:37.213688 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.213650 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b49c009-6df0-4641-b2ac-1f55403b6478-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn\" (UID: \"7b49c009-6df0-4641-b2ac-1f55403b6478\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" Sep 11 14:04:37.213864 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.213696 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b49c009-6df0-4641-b2ac-1f55403b6478-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn\" (UID: \"7b49c009-6df0-4641-b2ac-1f55403b6478\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" Sep 11 14:04:37.213906 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.213871 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jrjh\" (UniqueName: \"kubernetes.io/projected/7b49c009-6df0-4641-b2ac-1f55403b6478-kube-api-access-8jrjh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn\" (UID: \"7b49c009-6df0-4641-b2ac-1f55403b6478\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" Sep 11 14:04:37.315142 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.315106 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b49c009-6df0-4641-b2ac-1f55403b6478-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn\" (UID: \"7b49c009-6df0-4641-b2ac-1f55403b6478\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" Sep 11 14:04:37.315248 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.315151 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b49c009-6df0-4641-b2ac-1f55403b6478-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn\" (UID: \"7b49c009-6df0-4641-b2ac-1f55403b6478\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" Sep 11 14:04:37.315248 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.315193 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jrjh\" (UniqueName: \"kubernetes.io/projected/7b49c009-6df0-4641-b2ac-1f55403b6478-kube-api-access-8jrjh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn\" (UID: \"7b49c009-6df0-4641-b2ac-1f55403b6478\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" Sep 11 14:04:37.315541 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.315497 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b49c009-6df0-4641-b2ac-1f55403b6478-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn\" (UID: \"7b49c009-6df0-4641-b2ac-1f55403b6478\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" Sep 11 14:04:37.315541 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.315512 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b49c009-6df0-4641-b2ac-1f55403b6478-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn\" (UID: \"7b49c009-6df0-4641-b2ac-1f55403b6478\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" Sep 11 14:04:37.327577 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.327544 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jrjh\" (UniqueName: \"kubernetes.io/projected/7b49c009-6df0-4641-b2ac-1f55403b6478-kube-api-access-8jrjh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn\" (UID: \"7b49c009-6df0-4641-b2ac-1f55403b6478\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" Sep 11 14:04:37.379427 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.379391 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" Sep 11 14:04:37.648569 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.648498 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn"] Sep 11 14:04:37.651456 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:04:37.651430 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b49c009_6df0_4641_b2ac_1f55403b6478.slice/crio-ad060eca90fbb6c6fa0d87ebcc1c051722e34b776a7f09735730cede16e8e76a WatchSource:0}: Error finding container ad060eca90fbb6c6fa0d87ebcc1c051722e34b776a7f09735730cede16e8e76a: Status 404 returned error can't find the container with id ad060eca90fbb6c6fa0d87ebcc1c051722e34b776a7f09735730cede16e8e76a Sep 11 14:04:37.844506 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:37.844469 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" event={"ID":"7b49c009-6df0-4641-b2ac-1f55403b6478","Type":"ContainerStarted","Data":"ad060eca90fbb6c6fa0d87ebcc1c051722e34b776a7f09735730cede16e8e76a"} Sep 11 14:04:44.873487 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:44.873447 2566 generic.go:356] "Generic (PLEG): container finished" podID="7b49c009-6df0-4641-b2ac-1f55403b6478" containerID="412c8fa83ed9c1046bec1910f3aa47c5a705ad921dce2a02ba984a65e2f9348e" exitCode=0 Sep 11 14:04:44.873890 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:44.873495 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" event={"ID":"7b49c009-6df0-4641-b2ac-1f55403b6478","Type":"ContainerDied","Data":"412c8fa83ed9c1046bec1910f3aa47c5a705ad921dce2a02ba984a65e2f9348e"} Sep 11 14:04:47.888144 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:47.888105 2566 generic.go:356] "Generic (PLEG): container finished" podID="7b49c009-6df0-4641-b2ac-1f55403b6478" containerID="cdd17e47c139be6c2f5d6d11b99e5635ced5598774010748c7798e0782fcca22" exitCode=0 Sep 11 14:04:47.888542 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:47.888160 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" event={"ID":"7b49c009-6df0-4641-b2ac-1f55403b6478","Type":"ContainerDied","Data":"cdd17e47c139be6c2f5d6d11b99e5635ced5598774010748c7798e0782fcca22"} Sep 11 14:04:49.338060 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.338024 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d6c4d95fb-tvs2b"] Sep 11 14:04:49.341912 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.341885 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.352379 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.352317 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d6c4d95fb-tvs2b"] Sep 11 14:04:49.407077 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.407042 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da44b8e5-2f5a-446b-9aa3-f367dc008d81-service-ca\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.407286 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.407099 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da44b8e5-2f5a-446b-9aa3-f367dc008d81-oauth-serving-cert\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.407286 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.407149 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da44b8e5-2f5a-446b-9aa3-f367dc008d81-console-config\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.407286 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.407168 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da44b8e5-2f5a-446b-9aa3-f367dc008d81-console-oauth-config\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.407286 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.407260 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv8dc\" (UniqueName: \"kubernetes.io/projected/da44b8e5-2f5a-446b-9aa3-f367dc008d81-kube-api-access-hv8dc\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.407504 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.407312 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da44b8e5-2f5a-446b-9aa3-f367dc008d81-trusted-ca-bundle\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.407504 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.407343 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da44b8e5-2f5a-446b-9aa3-f367dc008d81-console-serving-cert\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.507941 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.507906 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da44b8e5-2f5a-446b-9aa3-f367dc008d81-console-config\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.507941 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.507940 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da44b8e5-2f5a-446b-9aa3-f367dc008d81-console-oauth-config\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.508199 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.507976 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv8dc\" (UniqueName: \"kubernetes.io/projected/da44b8e5-2f5a-446b-9aa3-f367dc008d81-kube-api-access-hv8dc\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.508199 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.508006 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da44b8e5-2f5a-446b-9aa3-f367dc008d81-trusted-ca-bundle\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.508199 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.508035 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da44b8e5-2f5a-446b-9aa3-f367dc008d81-console-serving-cert\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.508199 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.508071 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da44b8e5-2f5a-446b-9aa3-f367dc008d81-service-ca\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.508394 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.508242 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da44b8e5-2f5a-446b-9aa3-f367dc008d81-oauth-serving-cert\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.508717 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.508691 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da44b8e5-2f5a-446b-9aa3-f367dc008d81-console-config\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.508995 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.508862 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da44b8e5-2f5a-446b-9aa3-f367dc008d81-service-ca\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.508995 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.508928 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da44b8e5-2f5a-446b-9aa3-f367dc008d81-oauth-serving-cert\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.508995 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.508971 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da44b8e5-2f5a-446b-9aa3-f367dc008d81-trusted-ca-bundle\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.510591 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.510572 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da44b8e5-2f5a-446b-9aa3-f367dc008d81-console-oauth-config\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.510968 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.510951 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da44b8e5-2f5a-446b-9aa3-f367dc008d81-console-serving-cert\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.516825 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.516805 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv8dc\" (UniqueName: \"kubernetes.io/projected/da44b8e5-2f5a-446b-9aa3-f367dc008d81-kube-api-access-hv8dc\") pod \"console-7d6c4d95fb-tvs2b\" (UID: \"da44b8e5-2f5a-446b-9aa3-f367dc008d81\") " pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.653859 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.653767 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:49.950522 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:49.950445 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d6c4d95fb-tvs2b"] Sep 11 14:04:49.953140 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:04:49.953112 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda44b8e5_2f5a_446b_9aa3_f367dc008d81.slice/crio-b4216f94f4a80f06d1f24838d13f64682e93049a7ff015dd1fe873e7c6ba9e72 WatchSource:0}: Error finding container b4216f94f4a80f06d1f24838d13f64682e93049a7ff015dd1fe873e7c6ba9e72: Status 404 returned error can't find the container with id b4216f94f4a80f06d1f24838d13f64682e93049a7ff015dd1fe873e7c6ba9e72 Sep 11 14:04:50.902598 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:50.902552 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6c4d95fb-tvs2b" event={"ID":"da44b8e5-2f5a-446b-9aa3-f367dc008d81","Type":"ContainerStarted","Data":"8db7e00458cff562cb2e690702ecc3c92e4a2892c4eab8c3d746cded0971681f"} Sep 11 14:04:50.903046 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:50.902782 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d6c4d95fb-tvs2b" event={"ID":"da44b8e5-2f5a-446b-9aa3-f367dc008d81","Type":"ContainerStarted","Data":"b4216f94f4a80f06d1f24838d13f64682e93049a7ff015dd1fe873e7c6ba9e72"} Sep 11 14:04:50.926236 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:50.926190 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d6c4d95fb-tvs2b" podStartSLOduration=1.926175323 podStartE2EDuration="1.926175323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:04:50.925399114 +0000 UTC m=+431.330380243" watchObservedRunningTime="2026-09-11 14:04:50.926175323 +0000 UTC m=+431.331156450" Sep 11 14:04:54.921635 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:54.921594 2566 generic.go:356] "Generic (PLEG): container finished" podID="7b49c009-6df0-4641-b2ac-1f55403b6478" containerID="54f7407faa16fc7ba009f119b689cbf9adbe60666ca7f29e70f2c9a51b38656a" exitCode=0 Sep 11 14:04:54.922015 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:54.921673 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" event={"ID":"7b49c009-6df0-4641-b2ac-1f55403b6478","Type":"ContainerDied","Data":"54f7407faa16fc7ba009f119b689cbf9adbe60666ca7f29e70f2c9a51b38656a"} Sep 11 14:04:56.044640 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:56.044613 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" Sep 11 14:04:56.056875 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:56.056838 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b49c009-6df0-4641-b2ac-1f55403b6478-util\") pod \"7b49c009-6df0-4641-b2ac-1f55403b6478\" (UID: \"7b49c009-6df0-4641-b2ac-1f55403b6478\") " Sep 11 14:04:56.056994 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:56.056933 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jrjh\" (UniqueName: \"kubernetes.io/projected/7b49c009-6df0-4641-b2ac-1f55403b6478-kube-api-access-8jrjh\") pod \"7b49c009-6df0-4641-b2ac-1f55403b6478\" (UID: \"7b49c009-6df0-4641-b2ac-1f55403b6478\") " Sep 11 14:04:56.056994 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:56.056963 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b49c009-6df0-4641-b2ac-1f55403b6478-bundle\") pod \"7b49c009-6df0-4641-b2ac-1f55403b6478\" (UID: \"7b49c009-6df0-4641-b2ac-1f55403b6478\") " Sep 11 14:04:56.057829 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:56.057795 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b49c009-6df0-4641-b2ac-1f55403b6478-bundle" (OuterVolumeSpecName: "bundle") pod "7b49c009-6df0-4641-b2ac-1f55403b6478" (UID: "7b49c009-6df0-4641-b2ac-1f55403b6478"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:04:56.059267 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:56.059247 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b49c009-6df0-4641-b2ac-1f55403b6478-kube-api-access-8jrjh" (OuterVolumeSpecName: "kube-api-access-8jrjh") pod "7b49c009-6df0-4641-b2ac-1f55403b6478" (UID: "7b49c009-6df0-4641-b2ac-1f55403b6478"). InnerVolumeSpecName "kube-api-access-8jrjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:04:56.062384 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:56.062357 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b49c009-6df0-4641-b2ac-1f55403b6478-util" (OuterVolumeSpecName: "util") pod "7b49c009-6df0-4641-b2ac-1f55403b6478" (UID: "7b49c009-6df0-4641-b2ac-1f55403b6478"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:04:56.158213 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:56.158171 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jrjh\" (UniqueName: \"kubernetes.io/projected/7b49c009-6df0-4641-b2ac-1f55403b6478-kube-api-access-8jrjh\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:04:56.158213 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:56.158211 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b49c009-6df0-4641-b2ac-1f55403b6478-bundle\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:04:56.158411 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:56.158226 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b49c009-6df0-4641-b2ac-1f55403b6478-util\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:04:56.932016 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:56.931975 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" event={"ID":"7b49c009-6df0-4641-b2ac-1f55403b6478","Type":"ContainerDied","Data":"ad060eca90fbb6c6fa0d87ebcc1c051722e34b776a7f09735730cede16e8e76a"} Sep 11 14:04:56.932196 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:56.932053 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad060eca90fbb6c6fa0d87ebcc1c051722e34b776a7f09735730cede16e8e76a" Sep 11 14:04:56.932196 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:56.931994 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q4mrn" Sep 11 14:04:59.654752 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:59.654714 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:59.655233 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:59.654808 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:59.659593 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:59.659574 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:04:59.947651 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:04:59.947569 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d6c4d95fb-tvs2b" Sep 11 14:05:00.004856 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.004816 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c6c86cbd-j5stx"] Sep 11 14:05:00.633910 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.633861 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dkwg4"] Sep 11 14:05:00.634331 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.634309 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b49c009-6df0-4641-b2ac-1f55403b6478" containerName="extract" Sep 11 14:05:00.634331 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.634333 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b49c009-6df0-4641-b2ac-1f55403b6478" containerName="extract" Sep 11 14:05:00.634491 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.634347 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b49c009-6df0-4641-b2ac-1f55403b6478" containerName="util" Sep 11 14:05:00.634491 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.634355 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b49c009-6df0-4641-b2ac-1f55403b6478" containerName="util" Sep 11 14:05:00.634491 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.634365 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b49c009-6df0-4641-b2ac-1f55403b6478" containerName="pull" Sep 11 14:05:00.634491 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.634373 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b49c009-6df0-4641-b2ac-1f55403b6478" containerName="pull" Sep 11 14:05:00.634491 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.634452 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b49c009-6df0-4641-b2ac-1f55403b6478" containerName="extract" Sep 11 14:05:00.636473 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.636449 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dkwg4" Sep 11 14:05:00.639274 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.639250 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-6knxx\"" Sep 11 14:05:00.639380 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.639331 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:05:00.639436 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.639395 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 11 14:05:00.654706 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.654677 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dkwg4"] Sep 11 14:05:00.694486 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.694457 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4e42e1d-5efc-4e08-9123-5f0075d1486a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dkwg4\" (UID: \"a4e42e1d-5efc-4e08-9123-5f0075d1486a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dkwg4" Sep 11 14:05:00.694855 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.694495 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxrfb\" (UniqueName: \"kubernetes.io/projected/a4e42e1d-5efc-4e08-9123-5f0075d1486a-kube-api-access-vxrfb\") pod \"cert-manager-operator-controller-manager-55c444bf86-dkwg4\" (UID: \"a4e42e1d-5efc-4e08-9123-5f0075d1486a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dkwg4" Sep 11 14:05:00.794982 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.794938 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxrfb\" (UniqueName: \"kubernetes.io/projected/a4e42e1d-5efc-4e08-9123-5f0075d1486a-kube-api-access-vxrfb\") pod \"cert-manager-operator-controller-manager-55c444bf86-dkwg4\" (UID: \"a4e42e1d-5efc-4e08-9123-5f0075d1486a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dkwg4" Sep 11 14:05:00.795180 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.795029 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4e42e1d-5efc-4e08-9123-5f0075d1486a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dkwg4\" (UID: \"a4e42e1d-5efc-4e08-9123-5f0075d1486a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dkwg4" Sep 11 14:05:00.795364 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.795348 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4e42e1d-5efc-4e08-9123-5f0075d1486a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dkwg4\" (UID: \"a4e42e1d-5efc-4e08-9123-5f0075d1486a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dkwg4" Sep 11 14:05:00.812716 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.812682 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxrfb\" (UniqueName: \"kubernetes.io/projected/a4e42e1d-5efc-4e08-9123-5f0075d1486a-kube-api-access-vxrfb\") pod \"cert-manager-operator-controller-manager-55c444bf86-dkwg4\" (UID: \"a4e42e1d-5efc-4e08-9123-5f0075d1486a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dkwg4" Sep 11 14:05:00.944786 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:00.944707 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dkwg4" Sep 11 14:05:01.222143 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:01.222113 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dkwg4"] Sep 11 14:05:01.224858 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:05:01.224824 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e42e1d_5efc_4e08_9123_5f0075d1486a.slice/crio-c2bfa31235da4ebfe01d1db27ed6ad3d7790cb73f3b8d49f63b8160bad076a09 WatchSource:0}: Error finding container c2bfa31235da4ebfe01d1db27ed6ad3d7790cb73f3b8d49f63b8160bad076a09: Status 404 returned error can't find the container with id c2bfa31235da4ebfe01d1db27ed6ad3d7790cb73f3b8d49f63b8160bad076a09 Sep 11 14:05:01.953785 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:01.953742 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dkwg4" event={"ID":"a4e42e1d-5efc-4e08-9123-5f0075d1486a","Type":"ContainerStarted","Data":"c2bfa31235da4ebfe01d1db27ed6ad3d7790cb73f3b8d49f63b8160bad076a09"} Sep 11 14:05:02.960762 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:02.960719 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dkwg4" event={"ID":"a4e42e1d-5efc-4e08-9123-5f0075d1486a","Type":"ContainerStarted","Data":"e4b24b3755e28ee479fd7405630e31a76530aa0aa1159da2fef2b22c1849bf57"} Sep 11 14:05:02.990350 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:02.990298 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dkwg4" podStartSLOduration=1.456482665 podStartE2EDuration="2.990282651s" totalImagesPullingTime="1.533799986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:05:02.988938108 +0000 UTC m=+443.393919236" watchObservedRunningTime="2026-09-11 14:05:02.990282651 +0000 UTC m=+443.395263777" Sep 11 14:05:04.105692 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.105658 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb"] Sep 11 14:05:04.107909 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.107886 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" Sep 11 14:05:04.110573 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.110545 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:05:04.110573 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.110567 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:05:04.111465 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.111443 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-w8ql7\"" Sep 11 14:05:04.120077 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.120056 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb"] Sep 11 14:05:04.120625 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.120603 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb\" (UID: \"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" Sep 11 14:05:04.120741 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.120657 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb\" (UID: \"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" Sep 11 14:05:04.120741 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.120685 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj6g4\" (UniqueName: \"kubernetes.io/projected/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-kube-api-access-cj6g4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb\" (UID: \"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" Sep 11 14:05:04.221318 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.221281 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb\" (UID: \"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" Sep 11 14:05:04.221318 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.221317 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj6g4\" (UniqueName: \"kubernetes.io/projected/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-kube-api-access-cj6g4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb\" (UID: \"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" Sep 11 14:05:04.221510 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.221386 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb\" (UID: \"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" Sep 11 14:05:04.221723 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.221700 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb\" (UID: \"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" Sep 11 14:05:04.221756 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.221735 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb\" (UID: \"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" Sep 11 14:05:04.233756 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.233727 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj6g4\" (UniqueName: \"kubernetes.io/projected/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-kube-api-access-cj6g4\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb\" (UID: \"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" Sep 11 14:05:04.417415 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.417341 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" Sep 11 14:05:04.705487 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.705406 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb"] Sep 11 14:05:04.708469 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:05:04.708445 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee65b6c0_eff4_46b0_bb8d_ae80f01e2501.slice/crio-e674068728b445dad08b883d8f7e6ee9456a317d217ff27fb3638ea6954488f3 WatchSource:0}: Error finding container e674068728b445dad08b883d8f7e6ee9456a317d217ff27fb3638ea6954488f3: Status 404 returned error can't find the container with id e674068728b445dad08b883d8f7e6ee9456a317d217ff27fb3638ea6954488f3 Sep 11 14:05:04.813212 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.813185 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-h95mr"] Sep 11 14:05:04.814985 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.814969 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-h95mr" Sep 11 14:05:04.817646 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.817628 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 14:05:04.817716 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.817672 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 14:05:04.817964 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.817950 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vz57w\"" Sep 11 14:05:04.830765 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.830743 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-h95mr"] Sep 11 14:05:04.926392 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.926352 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e49a8b5-7c6a-4445-9aae-192e678e86cc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-h95mr\" (UID: \"0e49a8b5-7c6a-4445-9aae-192e678e86cc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h95mr" Sep 11 14:05:04.926392 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.926393 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78xv6\" (UniqueName: \"kubernetes.io/projected/0e49a8b5-7c6a-4445-9aae-192e678e86cc-kube-api-access-78xv6\") pod \"cert-manager-webhook-6486ffd997-h95mr\" (UID: \"0e49a8b5-7c6a-4445-9aae-192e678e86cc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h95mr" Sep 11 14:05:04.970156 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.970054 2566 generic.go:356] "Generic (PLEG): container finished" podID="ee65b6c0-eff4-46b0-bb8d-ae80f01e2501" containerID="401e3b9015daa80b1ee3d091c02409d98137955eeb2771ca027b9f9bf9f00dd2" exitCode=0 Sep 11 14:05:04.970156 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.970128 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" event={"ID":"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501","Type":"ContainerDied","Data":"401e3b9015daa80b1ee3d091c02409d98137955eeb2771ca027b9f9bf9f00dd2"} Sep 11 14:05:04.970365 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:04.970172 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" event={"ID":"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501","Type":"ContainerStarted","Data":"e674068728b445dad08b883d8f7e6ee9456a317d217ff27fb3638ea6954488f3"} Sep 11 14:05:05.027164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:05.027129 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e49a8b5-7c6a-4445-9aae-192e678e86cc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-h95mr\" (UID: \"0e49a8b5-7c6a-4445-9aae-192e678e86cc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h95mr" Sep 11 14:05:05.027342 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:05.027180 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78xv6\" (UniqueName: \"kubernetes.io/projected/0e49a8b5-7c6a-4445-9aae-192e678e86cc-kube-api-access-78xv6\") pod \"cert-manager-webhook-6486ffd997-h95mr\" (UID: \"0e49a8b5-7c6a-4445-9aae-192e678e86cc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h95mr" Sep 11 14:05:05.036258 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:05.036230 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e49a8b5-7c6a-4445-9aae-192e678e86cc-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-h95mr\" (UID: \"0e49a8b5-7c6a-4445-9aae-192e678e86cc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h95mr" Sep 11 14:05:05.036382 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:05.036365 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78xv6\" (UniqueName: \"kubernetes.io/projected/0e49a8b5-7c6a-4445-9aae-192e678e86cc-kube-api-access-78xv6\") pod \"cert-manager-webhook-6486ffd997-h95mr\" (UID: \"0e49a8b5-7c6a-4445-9aae-192e678e86cc\") " pod="cert-manager/cert-manager-webhook-6486ffd997-h95mr" Sep 11 14:05:05.163200 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:05.163162 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-h95mr" Sep 11 14:05:05.437013 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:05.436990 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-h95mr"] Sep 11 14:05:05.438685 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:05:05.438656 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e49a8b5_7c6a_4445_9aae_192e678e86cc.slice/crio-9e9202b330fe0bdc6031ddce9f93e5c915d3adefb8d2a6c248086441da9f769e WatchSource:0}: Error finding container 9e9202b330fe0bdc6031ddce9f93e5c915d3adefb8d2a6c248086441da9f769e: Status 404 returned error can't find the container with id 9e9202b330fe0bdc6031ddce9f93e5c915d3adefb8d2a6c248086441da9f769e Sep 11 14:05:05.975312 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:05.975277 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-h95mr" event={"ID":"0e49a8b5-7c6a-4445-9aae-192e678e86cc","Type":"ContainerStarted","Data":"9e9202b330fe0bdc6031ddce9f93e5c915d3adefb8d2a6c248086441da9f769e"} Sep 11 14:05:07.107858 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:07.107819 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fqdbq"] Sep 11 14:05:07.113780 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:07.113756 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fqdbq" Sep 11 14:05:07.117275 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:07.117248 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-dvbhp\"" Sep 11 14:05:07.121162 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:07.121138 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fqdbq"] Sep 11 14:05:07.145075 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:07.145052 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a59f8580-e7f1-4b1b-8929-ab7d1b58ff1d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fqdbq\" (UID: \"a59f8580-e7f1-4b1b-8929-ab7d1b58ff1d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fqdbq" Sep 11 14:05:07.145235 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:07.145113 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr4fp\" (UniqueName: \"kubernetes.io/projected/a59f8580-e7f1-4b1b-8929-ab7d1b58ff1d-kube-api-access-sr4fp\") pod \"cert-manager-cainjector-559cdd8c87-fqdbq\" (UID: \"a59f8580-e7f1-4b1b-8929-ab7d1b58ff1d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fqdbq" Sep 11 14:05:07.246491 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:07.246437 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a59f8580-e7f1-4b1b-8929-ab7d1b58ff1d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fqdbq\" (UID: \"a59f8580-e7f1-4b1b-8929-ab7d1b58ff1d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fqdbq" Sep 11 14:05:07.246491 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:07.246487 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr4fp\" (UniqueName: \"kubernetes.io/projected/a59f8580-e7f1-4b1b-8929-ab7d1b58ff1d-kube-api-access-sr4fp\") pod \"cert-manager-cainjector-559cdd8c87-fqdbq\" (UID: \"a59f8580-e7f1-4b1b-8929-ab7d1b58ff1d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fqdbq" Sep 11 14:05:07.255752 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:07.255721 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a59f8580-e7f1-4b1b-8929-ab7d1b58ff1d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fqdbq\" (UID: \"a59f8580-e7f1-4b1b-8929-ab7d1b58ff1d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fqdbq" Sep 11 14:05:07.255945 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:07.255923 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr4fp\" (UniqueName: \"kubernetes.io/projected/a59f8580-e7f1-4b1b-8929-ab7d1b58ff1d-kube-api-access-sr4fp\") pod \"cert-manager-cainjector-559cdd8c87-fqdbq\" (UID: \"a59f8580-e7f1-4b1b-8929-ab7d1b58ff1d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fqdbq" Sep 11 14:05:07.426255 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:07.426176 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fqdbq" Sep 11 14:05:08.639340 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:08.639308 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fqdbq"] Sep 11 14:05:08.641713 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:05:08.641685 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda59f8580_e7f1_4b1b_8929_ab7d1b58ff1d.slice/crio-0fa0bba0ca5d3333bed35f7011b5365431dfb143fb8d1f62d4287c084c08f1ac WatchSource:0}: Error finding container 0fa0bba0ca5d3333bed35f7011b5365431dfb143fb8d1f62d4287c084c08f1ac: Status 404 returned error can't find the container with id 0fa0bba0ca5d3333bed35f7011b5365431dfb143fb8d1f62d4287c084c08f1ac Sep 11 14:05:08.992818 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:08.992778 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-h95mr" event={"ID":"0e49a8b5-7c6a-4445-9aae-192e678e86cc","Type":"ContainerStarted","Data":"f0b79d5e1fdb7f9454168d48285f440e976c41e2be8ce06959df9e66c0df24c1"} Sep 11 14:05:08.992818 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:08.992834 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-h95mr" Sep 11 14:05:08.994452 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:08.994419 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fqdbq" event={"ID":"a59f8580-e7f1-4b1b-8929-ab7d1b58ff1d","Type":"ContainerStarted","Data":"b2d8905c84904dbbd6d14b783522f51f5ffeab8d45a9f340139f85b850e594ea"} Sep 11 14:05:08.994600 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:08.994462 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fqdbq" event={"ID":"a59f8580-e7f1-4b1b-8929-ab7d1b58ff1d","Type":"ContainerStarted","Data":"0fa0bba0ca5d3333bed35f7011b5365431dfb143fb8d1f62d4287c084c08f1ac"} Sep 11 14:05:08.996429 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:08.996371 2566 generic.go:356] "Generic (PLEG): container finished" podID="ee65b6c0-eff4-46b0-bb8d-ae80f01e2501" containerID="1dbeb6e354675f474af19433ce074d36420cb12bf8ef25876a396c1eb9e05db4" exitCode=0 Sep 11 14:05:08.996559 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:08.996511 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" event={"ID":"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501","Type":"ContainerDied","Data":"1dbeb6e354675f474af19433ce074d36420cb12bf8ef25876a396c1eb9e05db4"} Sep 11 14:05:09.014373 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:09.014333 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-h95mr" podStartSLOduration=2.078227971 podStartE2EDuration="5.014318369s" totalImagesPullingTime="2.936090398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:05:09.012399595 +0000 UTC m=+449.417380724" watchObservedRunningTime="2026-09-11 14:05:09.014318369 +0000 UTC m=+449.419299496" Sep 11 14:05:09.073139 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:09.073048 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fqdbq" podStartSLOduration=2.073034008 podStartE2EDuration="2.073034008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:05:09.072993016 +0000 UTC m=+449.477974145" watchObservedRunningTime="2026-09-11 14:05:09.073034008 +0000 UTC m=+449.478015135" Sep 11 14:05:10.004460 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:10.004420 2566 generic.go:356] "Generic (PLEG): container finished" podID="ee65b6c0-eff4-46b0-bb8d-ae80f01e2501" containerID="2eb6c05eed3b65e41ba8cc72c5d646cbe48e8fcd72d73e0e70a0d730df8c725b" exitCode=0 Sep 11 14:05:10.004947 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:10.004508 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" event={"ID":"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501","Type":"ContainerDied","Data":"2eb6c05eed3b65e41ba8cc72c5d646cbe48e8fcd72d73e0e70a0d730df8c725b"} Sep 11 14:05:11.130180 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:11.130151 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" Sep 11 14:05:11.178819 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:11.178778 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj6g4\" (UniqueName: \"kubernetes.io/projected/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-kube-api-access-cj6g4\") pod \"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501\" (UID: \"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501\") " Sep 11 14:05:11.178980 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:11.178877 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-bundle\") pod \"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501\" (UID: \"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501\") " Sep 11 14:05:11.178980 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:11.178944 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-util\") pod \"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501\" (UID: \"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501\") " Sep 11 14:05:11.179299 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:11.179270 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-bundle" (OuterVolumeSpecName: "bundle") pod "ee65b6c0-eff4-46b0-bb8d-ae80f01e2501" (UID: "ee65b6c0-eff4-46b0-bb8d-ae80f01e2501"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:05:11.181110 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:11.181069 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-kube-api-access-cj6g4" (OuterVolumeSpecName: "kube-api-access-cj6g4") pod "ee65b6c0-eff4-46b0-bb8d-ae80f01e2501" (UID: "ee65b6c0-eff4-46b0-bb8d-ae80f01e2501"). InnerVolumeSpecName "kube-api-access-cj6g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:05:11.185922 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:11.185885 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-util" (OuterVolumeSpecName: "util") pod "ee65b6c0-eff4-46b0-bb8d-ae80f01e2501" (UID: "ee65b6c0-eff4-46b0-bb8d-ae80f01e2501"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:05:11.280498 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:11.280408 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj6g4\" (UniqueName: \"kubernetes.io/projected/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-kube-api-access-cj6g4\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:11.280498 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:11.280444 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-bundle\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:11.280498 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:11.280457 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee65b6c0-eff4-46b0-bb8d-ae80f01e2501-util\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:12.015480 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:12.015441 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" event={"ID":"ee65b6c0-eff4-46b0-bb8d-ae80f01e2501","Type":"ContainerDied","Data":"e674068728b445dad08b883d8f7e6ee9456a317d217ff27fb3638ea6954488f3"} Sep 11 14:05:12.015480 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:12.015460 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptqgb" Sep 11 14:05:12.015730 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:12.015512 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e674068728b445dad08b883d8f7e6ee9456a317d217ff27fb3638ea6954488f3" Sep 11 14:05:14.822896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:14.822860 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8hmnr"] Sep 11 14:05:14.823367 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:14.823159 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee65b6c0-eff4-46b0-bb8d-ae80f01e2501" containerName="extract" Sep 11 14:05:14.823367 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:14.823170 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee65b6c0-eff4-46b0-bb8d-ae80f01e2501" containerName="extract" Sep 11 14:05:14.823367 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:14.823185 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee65b6c0-eff4-46b0-bb8d-ae80f01e2501" containerName="util" Sep 11 14:05:14.823367 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:14.823190 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee65b6c0-eff4-46b0-bb8d-ae80f01e2501" containerName="util" Sep 11 14:05:14.823367 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:14.823198 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee65b6c0-eff4-46b0-bb8d-ae80f01e2501" containerName="pull" Sep 11 14:05:14.823367 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:14.823203 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee65b6c0-eff4-46b0-bb8d-ae80f01e2501" containerName="pull" Sep 11 14:05:14.823367 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:14.823252 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee65b6c0-eff4-46b0-bb8d-ae80f01e2501" containerName="extract" Sep 11 14:05:14.825055 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:14.825039 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-8hmnr" Sep 11 14:05:14.828305 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:14.828281 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jzwrs\"" Sep 11 14:05:14.833795 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:14.833774 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8hmnr"] Sep 11 14:05:14.908890 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:14.908857 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdxkd\" (UniqueName: \"kubernetes.io/projected/b5cf6970-0a17-410b-ad7d-d03068b9102a-kube-api-access-bdxkd\") pod \"cert-manager-55b4d954d5-8hmnr\" (UID: \"b5cf6970-0a17-410b-ad7d-d03068b9102a\") " pod="cert-manager/cert-manager-55b4d954d5-8hmnr" Sep 11 14:05:14.909145 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:14.908934 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5cf6970-0a17-410b-ad7d-d03068b9102a-bound-sa-token\") pod \"cert-manager-55b4d954d5-8hmnr\" (UID: \"b5cf6970-0a17-410b-ad7d-d03068b9102a\") " pod="cert-manager/cert-manager-55b4d954d5-8hmnr" Sep 11 14:05:15.007306 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:15.007279 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-h95mr" Sep 11 14:05:15.009920 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:15.009900 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdxkd\" (UniqueName: \"kubernetes.io/projected/b5cf6970-0a17-410b-ad7d-d03068b9102a-kube-api-access-bdxkd\") pod \"cert-manager-55b4d954d5-8hmnr\" (UID: \"b5cf6970-0a17-410b-ad7d-d03068b9102a\") " pod="cert-manager/cert-manager-55b4d954d5-8hmnr" Sep 11 14:05:15.009978 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:15.009966 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5cf6970-0a17-410b-ad7d-d03068b9102a-bound-sa-token\") pod \"cert-manager-55b4d954d5-8hmnr\" (UID: \"b5cf6970-0a17-410b-ad7d-d03068b9102a\") " pod="cert-manager/cert-manager-55b4d954d5-8hmnr" Sep 11 14:05:15.020576 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:15.020548 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5cf6970-0a17-410b-ad7d-d03068b9102a-bound-sa-token\") pod \"cert-manager-55b4d954d5-8hmnr\" (UID: \"b5cf6970-0a17-410b-ad7d-d03068b9102a\") " pod="cert-manager/cert-manager-55b4d954d5-8hmnr" Sep 11 14:05:15.020718 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:15.020700 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdxkd\" (UniqueName: \"kubernetes.io/projected/b5cf6970-0a17-410b-ad7d-d03068b9102a-kube-api-access-bdxkd\") pod \"cert-manager-55b4d954d5-8hmnr\" (UID: \"b5cf6970-0a17-410b-ad7d-d03068b9102a\") " pod="cert-manager/cert-manager-55b4d954d5-8hmnr" Sep 11 14:05:15.134016 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:15.133934 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-8hmnr" Sep 11 14:05:15.409042 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:15.408970 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-8hmnr"] Sep 11 14:05:15.412353 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:05:15.412320 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5cf6970_0a17_410b_ad7d_d03068b9102a.slice/crio-fe558eaea332165ed155c0bfb35d1a37abfd7178b0e45cff2fc93453e8dfd839 WatchSource:0}: Error finding container fe558eaea332165ed155c0bfb35d1a37abfd7178b0e45cff2fc93453e8dfd839: Status 404 returned error can't find the container with id fe558eaea332165ed155c0bfb35d1a37abfd7178b0e45cff2fc93453e8dfd839 Sep 11 14:05:16.034970 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:16.034938 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-8hmnr" event={"ID":"b5cf6970-0a17-410b-ad7d-d03068b9102a","Type":"ContainerStarted","Data":"052e741adb8e0f10c92db1582499d2a025bad6f701b24d85eb2c6cd558fc9256"} Sep 11 14:05:16.035352 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:16.034978 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-8hmnr" event={"ID":"b5cf6970-0a17-410b-ad7d-d03068b9102a","Type":"ContainerStarted","Data":"fe558eaea332165ed155c0bfb35d1a37abfd7178b0e45cff2fc93453e8dfd839"} Sep 11 14:05:16.076981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:16.076920 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-8hmnr" podStartSLOduration=2.076904172 podStartE2EDuration="2.076904172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:05:16.076022336 +0000 UTC m=+456.481003464" watchObservedRunningTime="2026-09-11 14:05:16.076904172 +0000 UTC m=+456.481885300" Sep 11 14:05:25.024578 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.024536 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c6c86cbd-j5stx" podUID="1f7fac22-3817-4831-a873-a1b09ef35696" containerName="console" containerID="cri-o://246dbae5b19c11bc2b4d2260b0294322f4dbb176d5df81afa53047e7bb9591fe" gracePeriod=15 Sep 11 14:05:25.271567 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.271543 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:05:25.393071 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.392972 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f7fac22-3817-4831-a873-a1b09ef35696-console-serving-cert\") pod \"1f7fac22-3817-4831-a873-a1b09ef35696\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " Sep 11 14:05:25.393071 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.393026 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-oauth-serving-cert\") pod \"1f7fac22-3817-4831-a873-a1b09ef35696\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " Sep 11 14:05:25.393071 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.393047 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-trusted-ca-bundle\") pod \"1f7fac22-3817-4831-a873-a1b09ef35696\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " Sep 11 14:05:25.393375 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.393101 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-console-config\") pod \"1f7fac22-3817-4831-a873-a1b09ef35696\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " Sep 11 14:05:25.393375 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.393126 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f7fac22-3817-4831-a873-a1b09ef35696-console-oauth-config\") pod \"1f7fac22-3817-4831-a873-a1b09ef35696\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " Sep 11 14:05:25.393375 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.393155 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-service-ca\") pod \"1f7fac22-3817-4831-a873-a1b09ef35696\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " Sep 11 14:05:25.393375 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.393184 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-866s2\" (UniqueName: \"kubernetes.io/projected/1f7fac22-3817-4831-a873-a1b09ef35696-kube-api-access-866s2\") pod \"1f7fac22-3817-4831-a873-a1b09ef35696\" (UID: \"1f7fac22-3817-4831-a873-a1b09ef35696\") " Sep 11 14:05:25.393582 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.393548 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1f7fac22-3817-4831-a873-a1b09ef35696" (UID: "1f7fac22-3817-4831-a873-a1b09ef35696"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:05:25.393641 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.393570 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-console-config" (OuterVolumeSpecName: "console-config") pod "1f7fac22-3817-4831-a873-a1b09ef35696" (UID: "1f7fac22-3817-4831-a873-a1b09ef35696"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:05:25.393641 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.393578 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1f7fac22-3817-4831-a873-a1b09ef35696" (UID: "1f7fac22-3817-4831-a873-a1b09ef35696"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:05:25.393641 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.393589 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-service-ca" (OuterVolumeSpecName: "service-ca") pod "1f7fac22-3817-4831-a873-a1b09ef35696" (UID: "1f7fac22-3817-4831-a873-a1b09ef35696"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:05:25.395415 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.395391 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7fac22-3817-4831-a873-a1b09ef35696-kube-api-access-866s2" (OuterVolumeSpecName: "kube-api-access-866s2") pod "1f7fac22-3817-4831-a873-a1b09ef35696" (UID: "1f7fac22-3817-4831-a873-a1b09ef35696"). InnerVolumeSpecName "kube-api-access-866s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:05:25.395533 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.395488 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7fac22-3817-4831-a873-a1b09ef35696-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1f7fac22-3817-4831-a873-a1b09ef35696" (UID: "1f7fac22-3817-4831-a873-a1b09ef35696"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:05:25.395672 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.395546 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f7fac22-3817-4831-a873-a1b09ef35696-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1f7fac22-3817-4831-a873-a1b09ef35696" (UID: "1f7fac22-3817-4831-a873-a1b09ef35696"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:05:25.494553 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.494507 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-console-config\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:25.494553 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.494543 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f7fac22-3817-4831-a873-a1b09ef35696-console-oauth-config\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:25.494553 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.494557 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:25.494799 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.494568 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-866s2\" (UniqueName: \"kubernetes.io/projected/1f7fac22-3817-4831-a873-a1b09ef35696-kube-api-access-866s2\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:25.494799 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.494580 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f7fac22-3817-4831-a873-a1b09ef35696-console-serving-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:25.494799 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.494593 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-oauth-serving-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:25.494799 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:25.494604 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f7fac22-3817-4831-a873-a1b09ef35696-trusted-ca-bundle\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:26.081656 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:26.081617 2566 generic.go:356] "Generic (PLEG): container finished" podID="1f7fac22-3817-4831-a873-a1b09ef35696" containerID="246dbae5b19c11bc2b4d2260b0294322f4dbb176d5df81afa53047e7bb9591fe" exitCode=0 Sep 11 14:05:26.082052 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:26.081671 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6c86cbd-j5stx" Sep 11 14:05:26.082052 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:26.081711 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6c86cbd-j5stx" event={"ID":"1f7fac22-3817-4831-a873-a1b09ef35696","Type":"ContainerDied","Data":"246dbae5b19c11bc2b4d2260b0294322f4dbb176d5df81afa53047e7bb9591fe"} Sep 11 14:05:26.082052 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:26.081762 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6c86cbd-j5stx" event={"ID":"1f7fac22-3817-4831-a873-a1b09ef35696","Type":"ContainerDied","Data":"668282fb09a34144c74e10714614bab1c2d3217694a7aacc742e0262658d33b2"} Sep 11 14:05:26.082052 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:26.081783 2566 scope.go:117] "RemoveContainer" containerID="246dbae5b19c11bc2b4d2260b0294322f4dbb176d5df81afa53047e7bb9591fe" Sep 11 14:05:26.090229 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:26.090209 2566 scope.go:117] "RemoveContainer" containerID="246dbae5b19c11bc2b4d2260b0294322f4dbb176d5df81afa53047e7bb9591fe" Sep 11 14:05:26.090506 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:05:26.090487 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246dbae5b19c11bc2b4d2260b0294322f4dbb176d5df81afa53047e7bb9591fe\": container with ID starting with 246dbae5b19c11bc2b4d2260b0294322f4dbb176d5df81afa53047e7bb9591fe not found: ID does not exist" containerID="246dbae5b19c11bc2b4d2260b0294322f4dbb176d5df81afa53047e7bb9591fe" Sep 11 14:05:26.090563 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:26.090519 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246dbae5b19c11bc2b4d2260b0294322f4dbb176d5df81afa53047e7bb9591fe"} err="failed to get container status \"246dbae5b19c11bc2b4d2260b0294322f4dbb176d5df81afa53047e7bb9591fe\": rpc error: code = NotFound desc = could not find container \"246dbae5b19c11bc2b4d2260b0294322f4dbb176d5df81afa53047e7bb9591fe\": container with ID starting with 246dbae5b19c11bc2b4d2260b0294322f4dbb176d5df81afa53047e7bb9591fe not found: ID does not exist" Sep 11 14:05:26.104887 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:26.104851 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c6c86cbd-j5stx"] Sep 11 14:05:26.107297 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:26.107274 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c6c86cbd-j5stx"] Sep 11 14:05:28.095954 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:28.095916 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f7fac22-3817-4831-a873-a1b09ef35696" path="/var/lib/kubelet/pods/1f7fac22-3817-4831-a873-a1b09ef35696/volumes" Sep 11 14:05:30.658361 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.658327 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh"] Sep 11 14:05:30.658777 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.658626 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7fac22-3817-4831-a873-a1b09ef35696" containerName="console" Sep 11 14:05:30.658777 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.658636 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7fac22-3817-4831-a873-a1b09ef35696" containerName="console" Sep 11 14:05:30.658777 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.658697 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f7fac22-3817-4831-a873-a1b09ef35696" containerName="console" Sep 11 14:05:30.680366 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.680336 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh"] Sep 11 14:05:30.680521 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.680441 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" Sep 11 14:05:30.683396 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.683357 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:05:30.683396 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.683395 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:05:30.684492 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.684474 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-w8ql7\"" Sep 11 14:05:30.740305 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.740273 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7qnw\" (UniqueName: \"kubernetes.io/projected/384a5e9d-9349-48bc-a944-8ebecffcedaf-kube-api-access-r7qnw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh\" (UID: \"384a5e9d-9349-48bc-a944-8ebecffcedaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" Sep 11 14:05:30.740519 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.740337 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/384a5e9d-9349-48bc-a944-8ebecffcedaf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh\" (UID: \"384a5e9d-9349-48bc-a944-8ebecffcedaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" Sep 11 14:05:30.740519 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.740359 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/384a5e9d-9349-48bc-a944-8ebecffcedaf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh\" (UID: \"384a5e9d-9349-48bc-a944-8ebecffcedaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" Sep 11 14:05:30.840801 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.840759 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/384a5e9d-9349-48bc-a944-8ebecffcedaf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh\" (UID: \"384a5e9d-9349-48bc-a944-8ebecffcedaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" Sep 11 14:05:30.840801 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.840802 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/384a5e9d-9349-48bc-a944-8ebecffcedaf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh\" (UID: \"384a5e9d-9349-48bc-a944-8ebecffcedaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" Sep 11 14:05:30.841043 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.840865 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7qnw\" (UniqueName: \"kubernetes.io/projected/384a5e9d-9349-48bc-a944-8ebecffcedaf-kube-api-access-r7qnw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh\" (UID: \"384a5e9d-9349-48bc-a944-8ebecffcedaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" Sep 11 14:05:30.841285 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.841261 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/384a5e9d-9349-48bc-a944-8ebecffcedaf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh\" (UID: \"384a5e9d-9349-48bc-a944-8ebecffcedaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" Sep 11 14:05:30.841326 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.841273 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/384a5e9d-9349-48bc-a944-8ebecffcedaf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh\" (UID: \"384a5e9d-9349-48bc-a944-8ebecffcedaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" Sep 11 14:05:30.875548 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.875522 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7qnw\" (UniqueName: \"kubernetes.io/projected/384a5e9d-9349-48bc-a944-8ebecffcedaf-kube-api-access-r7qnw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh\" (UID: \"384a5e9d-9349-48bc-a944-8ebecffcedaf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" Sep 11 14:05:30.989435 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:30.989409 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" Sep 11 14:05:31.269269 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:31.269246 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh"] Sep 11 14:05:31.271324 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:05:31.271297 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384a5e9d_9349_48bc_a944_8ebecffcedaf.slice/crio-a78be9c2c0a13f683a569c197f1184e3f6ab83a9a3703094edc8fbca4ba35ed0 WatchSource:0}: Error finding container a78be9c2c0a13f683a569c197f1184e3f6ab83a9a3703094edc8fbca4ba35ed0: Status 404 returned error can't find the container with id a78be9c2c0a13f683a569c197f1184e3f6ab83a9a3703094edc8fbca4ba35ed0 Sep 11 14:05:32.109666 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:32.109635 2566 generic.go:356] "Generic (PLEG): container finished" podID="384a5e9d-9349-48bc-a944-8ebecffcedaf" containerID="f214404c9fb39a338c428ffb32d7e7e54fe003b81f80b55603c3991ecc6c913b" exitCode=0 Sep 11 14:05:32.110051 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:32.109693 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" event={"ID":"384a5e9d-9349-48bc-a944-8ebecffcedaf","Type":"ContainerDied","Data":"f214404c9fb39a338c428ffb32d7e7e54fe003b81f80b55603c3991ecc6c913b"} Sep 11 14:05:32.110051 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:32.109729 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" event={"ID":"384a5e9d-9349-48bc-a944-8ebecffcedaf","Type":"ContainerStarted","Data":"a78be9c2c0a13f683a569c197f1184e3f6ab83a9a3703094edc8fbca4ba35ed0"} Sep 11 14:05:33.115641 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:33.115600 2566 generic.go:356] "Generic (PLEG): container finished" podID="384a5e9d-9349-48bc-a944-8ebecffcedaf" containerID="0abbd301d551a07b1ea89ec2443e76d49fbcf25a3b450a3875cded8a1eac2c77" exitCode=0 Sep 11 14:05:33.116071 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:33.115666 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" event={"ID":"384a5e9d-9349-48bc-a944-8ebecffcedaf","Type":"ContainerDied","Data":"0abbd301d551a07b1ea89ec2443e76d49fbcf25a3b450a3875cded8a1eac2c77"} Sep 11 14:05:34.121907 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:34.121875 2566 generic.go:356] "Generic (PLEG): container finished" podID="384a5e9d-9349-48bc-a944-8ebecffcedaf" containerID="0c8b590c1397a2ce408ad29e15f2db89d6dda84f40b735239a031b70d56ff04d" exitCode=0 Sep 11 14:05:34.122325 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:34.121961 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" event={"ID":"384a5e9d-9349-48bc-a944-8ebecffcedaf","Type":"ContainerDied","Data":"0c8b590c1397a2ce408ad29e15f2db89d6dda84f40b735239a031b70d56ff04d"} Sep 11 14:05:35.247019 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:35.246990 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" Sep 11 14:05:35.378665 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:35.378546 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/384a5e9d-9349-48bc-a944-8ebecffcedaf-util\") pod \"384a5e9d-9349-48bc-a944-8ebecffcedaf\" (UID: \"384a5e9d-9349-48bc-a944-8ebecffcedaf\") " Sep 11 14:05:35.378665 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:35.378645 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7qnw\" (UniqueName: \"kubernetes.io/projected/384a5e9d-9349-48bc-a944-8ebecffcedaf-kube-api-access-r7qnw\") pod \"384a5e9d-9349-48bc-a944-8ebecffcedaf\" (UID: \"384a5e9d-9349-48bc-a944-8ebecffcedaf\") " Sep 11 14:05:35.378900 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:35.378680 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/384a5e9d-9349-48bc-a944-8ebecffcedaf-bundle\") pod \"384a5e9d-9349-48bc-a944-8ebecffcedaf\" (UID: \"384a5e9d-9349-48bc-a944-8ebecffcedaf\") " Sep 11 14:05:35.379409 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:35.379383 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384a5e9d-9349-48bc-a944-8ebecffcedaf-bundle" (OuterVolumeSpecName: "bundle") pod "384a5e9d-9349-48bc-a944-8ebecffcedaf" (UID: "384a5e9d-9349-48bc-a944-8ebecffcedaf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:05:35.380880 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:35.380842 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384a5e9d-9349-48bc-a944-8ebecffcedaf-kube-api-access-r7qnw" (OuterVolumeSpecName: "kube-api-access-r7qnw") pod "384a5e9d-9349-48bc-a944-8ebecffcedaf" (UID: "384a5e9d-9349-48bc-a944-8ebecffcedaf"). InnerVolumeSpecName "kube-api-access-r7qnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:05:35.384069 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:35.384042 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384a5e9d-9349-48bc-a944-8ebecffcedaf-util" (OuterVolumeSpecName: "util") pod "384a5e9d-9349-48bc-a944-8ebecffcedaf" (UID: "384a5e9d-9349-48bc-a944-8ebecffcedaf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:05:35.479579 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:35.479538 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/384a5e9d-9349-48bc-a944-8ebecffcedaf-util\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:35.479579 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:35.479571 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7qnw\" (UniqueName: \"kubernetes.io/projected/384a5e9d-9349-48bc-a944-8ebecffcedaf-kube-api-access-r7qnw\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:35.479579 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:35.479582 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/384a5e9d-9349-48bc-a944-8ebecffcedaf-bundle\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:36.133195 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:36.133167 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" Sep 11 14:05:36.133346 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:36.133168 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbfnbh" event={"ID":"384a5e9d-9349-48bc-a944-8ebecffcedaf","Type":"ContainerDied","Data":"a78be9c2c0a13f683a569c197f1184e3f6ab83a9a3703094edc8fbca4ba35ed0"} Sep 11 14:05:36.133346 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:36.133291 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a78be9c2c0a13f683a569c197f1184e3f6ab83a9a3703094edc8fbca4ba35ed0" Sep 11 14:05:44.931906 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:44.931863 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb"] Sep 11 14:05:44.932389 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:44.932325 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="384a5e9d-9349-48bc-a944-8ebecffcedaf" containerName="pull" Sep 11 14:05:44.932389 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:44.932343 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="384a5e9d-9349-48bc-a944-8ebecffcedaf" containerName="pull" Sep 11 14:05:44.932389 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:44.932368 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="384a5e9d-9349-48bc-a944-8ebecffcedaf" containerName="extract" Sep 11 14:05:44.932389 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:44.932376 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="384a5e9d-9349-48bc-a944-8ebecffcedaf" containerName="extract" Sep 11 14:05:44.932602 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:44.932401 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="384a5e9d-9349-48bc-a944-8ebecffcedaf" containerName="util" Sep 11 14:05:44.932602 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:44.932409 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="384a5e9d-9349-48bc-a944-8ebecffcedaf" containerName="util" Sep 11 14:05:44.932602 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:44.932492 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="384a5e9d-9349-48bc-a944-8ebecffcedaf" containerName="extract" Sep 11 14:05:44.937325 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:44.937304 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" Sep 11 14:05:44.942233 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:44.942208 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:05:44.942374 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:44.942358 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-w8ql7\"" Sep 11 14:05:44.943062 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:44.943039 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:05:44.946629 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:44.946609 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb"] Sep 11 14:05:45.059647 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:45.059599 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c65048a-744b-454b-a887-1a39ae83a407-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb\" (UID: \"9c65048a-744b-454b-a887-1a39ae83a407\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" Sep 11 14:05:45.059647 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:45.059647 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c65048a-744b-454b-a887-1a39ae83a407-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb\" (UID: \"9c65048a-744b-454b-a887-1a39ae83a407\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" Sep 11 14:05:45.059864 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:45.059754 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmb2d\" (UniqueName: \"kubernetes.io/projected/9c65048a-744b-454b-a887-1a39ae83a407-kube-api-access-nmb2d\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb\" (UID: \"9c65048a-744b-454b-a887-1a39ae83a407\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" Sep 11 14:05:45.160695 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:45.160662 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmb2d\" (UniqueName: \"kubernetes.io/projected/9c65048a-744b-454b-a887-1a39ae83a407-kube-api-access-nmb2d\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb\" (UID: \"9c65048a-744b-454b-a887-1a39ae83a407\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" Sep 11 14:05:45.160868 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:45.160741 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c65048a-744b-454b-a887-1a39ae83a407-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb\" (UID: \"9c65048a-744b-454b-a887-1a39ae83a407\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" Sep 11 14:05:45.160868 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:45.160768 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c65048a-744b-454b-a887-1a39ae83a407-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb\" (UID: \"9c65048a-744b-454b-a887-1a39ae83a407\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" Sep 11 14:05:45.161150 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:45.161132 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c65048a-744b-454b-a887-1a39ae83a407-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb\" (UID: \"9c65048a-744b-454b-a887-1a39ae83a407\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" Sep 11 14:05:45.161203 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:45.161176 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c65048a-744b-454b-a887-1a39ae83a407-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb\" (UID: \"9c65048a-744b-454b-a887-1a39ae83a407\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" Sep 11 14:05:45.173626 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:45.173598 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmb2d\" (UniqueName: \"kubernetes.io/projected/9c65048a-744b-454b-a887-1a39ae83a407-kube-api-access-nmb2d\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb\" (UID: \"9c65048a-744b-454b-a887-1a39ae83a407\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" Sep 11 14:05:45.247191 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:45.247152 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" Sep 11 14:05:45.521855 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:45.521828 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb"] Sep 11 14:05:45.524148 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:05:45.524123 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c65048a_744b_454b_a887_1a39ae83a407.slice/crio-dbbcebd48189a1cca1ecb3b024f8851e0a4899857d7d8d9cc3226e2e8326b2cb WatchSource:0}: Error finding container dbbcebd48189a1cca1ecb3b024f8851e0a4899857d7d8d9cc3226e2e8326b2cb: Status 404 returned error can't find the container with id dbbcebd48189a1cca1ecb3b024f8851e0a4899857d7d8d9cc3226e2e8326b2cb Sep 11 14:05:46.177481 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.177450 2566 generic.go:356] "Generic (PLEG): container finished" podID="9c65048a-744b-454b-a887-1a39ae83a407" containerID="7868ef151c5577184577f99cf8db96ce702b0cc2afb9adb07f19093916efded3" exitCode=0 Sep 11 14:05:46.177837 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.177546 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" event={"ID":"9c65048a-744b-454b-a887-1a39ae83a407","Type":"ContainerDied","Data":"7868ef151c5577184577f99cf8db96ce702b0cc2afb9adb07f19093916efded3"} Sep 11 14:05:46.177837 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.177595 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" event={"ID":"9c65048a-744b-454b-a887-1a39ae83a407","Type":"ContainerStarted","Data":"dbbcebd48189a1cca1ecb3b024f8851e0a4899857d7d8d9cc3226e2e8326b2cb"} Sep 11 14:05:46.789790 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.789763 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n"] Sep 11 14:05:46.792958 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.792938 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:46.795733 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.795712 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 14:05:46.795852 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.795756 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 14:05:46.795852 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.795796 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 14:05:46.795852 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.795757 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 14:05:46.795852 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.795848 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2hc6w\"" Sep 11 14:05:46.813651 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.813628 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n"] Sep 11 14:05:46.976780 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.976718 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8119fc8-fece-463f-8abb-c549cbbb0126-manifests\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-z2w7n\" (UID: \"b8119fc8-fece-463f-8abb-c549cbbb0126\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:46.976780 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.976785 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8119fc8-fece-463f-8abb-c549cbbb0126-webhook-cert\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-z2w7n\" (UID: \"b8119fc8-fece-463f-8abb-c549cbbb0126\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:46.977005 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.976878 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddb8r\" (UniqueName: \"kubernetes.io/projected/b8119fc8-fece-463f-8abb-c549cbbb0126-kube-api-access-ddb8r\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-z2w7n\" (UID: \"b8119fc8-fece-463f-8abb-c549cbbb0126\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:46.977005 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.976964 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8119fc8-fece-463f-8abb-c549cbbb0126-tmp\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-z2w7n\" (UID: \"b8119fc8-fece-463f-8abb-c549cbbb0126\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:46.977005 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:46.976987 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8119fc8-fece-463f-8abb-c549cbbb0126-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-z2w7n\" (UID: \"b8119fc8-fece-463f-8abb-c549cbbb0126\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:47.077853 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:47.077763 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8119fc8-fece-463f-8abb-c549cbbb0126-tmp\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-z2w7n\" (UID: \"b8119fc8-fece-463f-8abb-c549cbbb0126\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:47.077853 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:47.077803 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8119fc8-fece-463f-8abb-c549cbbb0126-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-z2w7n\" (UID: \"b8119fc8-fece-463f-8abb-c549cbbb0126\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:47.077853 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:47.077833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8119fc8-fece-463f-8abb-c549cbbb0126-manifests\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-z2w7n\" (UID: \"b8119fc8-fece-463f-8abb-c549cbbb0126\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:47.077853 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:47.077849 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8119fc8-fece-463f-8abb-c549cbbb0126-webhook-cert\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-z2w7n\" (UID: \"b8119fc8-fece-463f-8abb-c549cbbb0126\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:47.078200 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:47.077876 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddb8r\" (UniqueName: \"kubernetes.io/projected/b8119fc8-fece-463f-8abb-c549cbbb0126-kube-api-access-ddb8r\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-z2w7n\" (UID: \"b8119fc8-fece-463f-8abb-c549cbbb0126\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:47.078278 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:47.078251 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b8119fc8-fece-463f-8abb-c549cbbb0126-manifests\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-z2w7n\" (UID: \"b8119fc8-fece-463f-8abb-c549cbbb0126\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:47.080355 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:47.080329 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8119fc8-fece-463f-8abb-c549cbbb0126-tmp\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-z2w7n\" (UID: \"b8119fc8-fece-463f-8abb-c549cbbb0126\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:47.080672 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:47.080651 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b8119fc8-fece-463f-8abb-c549cbbb0126-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-z2w7n\" (UID: \"b8119fc8-fece-463f-8abb-c549cbbb0126\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:47.080713 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:47.080651 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8119fc8-fece-463f-8abb-c549cbbb0126-webhook-cert\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-z2w7n\" (UID: \"b8119fc8-fece-463f-8abb-c549cbbb0126\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:47.090324 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:47.090298 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddb8r\" (UniqueName: \"kubernetes.io/projected/b8119fc8-fece-463f-8abb-c549cbbb0126-kube-api-access-ddb8r\") pod \"opendatahub-operator-controller-manager-5586bcdcf6-z2w7n\" (UID: \"b8119fc8-fece-463f-8abb-c549cbbb0126\") " pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:47.104138 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:47.104112 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:47.185246 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:47.185216 2566 generic.go:356] "Generic (PLEG): container finished" podID="9c65048a-744b-454b-a887-1a39ae83a407" containerID="21e17d540279dadcce940a22e2f29c7bef4e7ce0c7667885b44b3814288342c5" exitCode=0 Sep 11 14:05:47.185671 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:47.185305 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" event={"ID":"9c65048a-744b-454b-a887-1a39ae83a407","Type":"ContainerDied","Data":"21e17d540279dadcce940a22e2f29c7bef4e7ce0c7667885b44b3814288342c5"} Sep 11 14:05:47.402548 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:47.402518 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n"] Sep 11 14:05:47.411072 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:05:47.411047 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8119fc8_fece_463f_8abb_c549cbbb0126.slice/crio-765d11a051ae4c4a4f2b2d6ab0038b04c1fa27a5d4dc8765d7beb165579a53c1 WatchSource:0}: Error finding container 765d11a051ae4c4a4f2b2d6ab0038b04c1fa27a5d4dc8765d7beb165579a53c1: Status 404 returned error can't find the container with id 765d11a051ae4c4a4f2b2d6ab0038b04c1fa27a5d4dc8765d7beb165579a53c1 Sep 11 14:05:48.194388 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:48.194349 2566 generic.go:356] "Generic (PLEG): container finished" podID="9c65048a-744b-454b-a887-1a39ae83a407" containerID="b048767806d5d0ce4e22ff0faa676c92b553e1a9c96ff64ca6f566ef451038b4" exitCode=0 Sep 11 14:05:48.194856 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:48.194445 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" event={"ID":"9c65048a-744b-454b-a887-1a39ae83a407","Type":"ContainerDied","Data":"b048767806d5d0ce4e22ff0faa676c92b553e1a9c96ff64ca6f566ef451038b4"} Sep 11 14:05:48.196410 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:48.196375 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" event={"ID":"b8119fc8-fece-463f-8abb-c549cbbb0126","Type":"ContainerStarted","Data":"765d11a051ae4c4a4f2b2d6ab0038b04c1fa27a5d4dc8765d7beb165579a53c1"} Sep 11 14:05:50.547538 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:50.547510 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" Sep 11 14:05:50.708955 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:50.708924 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmb2d\" (UniqueName: \"kubernetes.io/projected/9c65048a-744b-454b-a887-1a39ae83a407-kube-api-access-nmb2d\") pod \"9c65048a-744b-454b-a887-1a39ae83a407\" (UID: \"9c65048a-744b-454b-a887-1a39ae83a407\") " Sep 11 14:05:50.708955 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:50.708961 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c65048a-744b-454b-a887-1a39ae83a407-bundle\") pod \"9c65048a-744b-454b-a887-1a39ae83a407\" (UID: \"9c65048a-744b-454b-a887-1a39ae83a407\") " Sep 11 14:05:50.709182 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:50.709031 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c65048a-744b-454b-a887-1a39ae83a407-util\") pod \"9c65048a-744b-454b-a887-1a39ae83a407\" (UID: \"9c65048a-744b-454b-a887-1a39ae83a407\") " Sep 11 14:05:50.709777 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:50.709746 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c65048a-744b-454b-a887-1a39ae83a407-bundle" (OuterVolumeSpecName: "bundle") pod "9c65048a-744b-454b-a887-1a39ae83a407" (UID: "9c65048a-744b-454b-a887-1a39ae83a407"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:05:50.711241 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:50.711219 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c65048a-744b-454b-a887-1a39ae83a407-kube-api-access-nmb2d" (OuterVolumeSpecName: "kube-api-access-nmb2d") pod "9c65048a-744b-454b-a887-1a39ae83a407" (UID: "9c65048a-744b-454b-a887-1a39ae83a407"). InnerVolumeSpecName "kube-api-access-nmb2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:05:50.714139 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:50.714114 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c65048a-744b-454b-a887-1a39ae83a407-util" (OuterVolumeSpecName: "util") pod "9c65048a-744b-454b-a887-1a39ae83a407" (UID: "9c65048a-744b-454b-a887-1a39ae83a407"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:05:50.809616 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:50.809542 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c65048a-744b-454b-a887-1a39ae83a407-util\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:50.809616 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:50.809569 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmb2d\" (UniqueName: \"kubernetes.io/projected/9c65048a-744b-454b-a887-1a39ae83a407-kube-api-access-nmb2d\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:50.809616 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:50.809579 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c65048a-744b-454b-a887-1a39ae83a407-bundle\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:05:51.213716 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:51.213646 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" Sep 11 14:05:51.213871 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:51.213645 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gbxbb" event={"ID":"9c65048a-744b-454b-a887-1a39ae83a407","Type":"ContainerDied","Data":"dbbcebd48189a1cca1ecb3b024f8851e0a4899857d7d8d9cc3226e2e8326b2cb"} Sep 11 14:05:51.213871 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:51.213781 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbbcebd48189a1cca1ecb3b024f8851e0a4899857d7d8d9cc3226e2e8326b2cb" Sep 11 14:05:51.215467 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:51.215439 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" event={"ID":"b8119fc8-fece-463f-8abb-c549cbbb0126","Type":"ContainerStarted","Data":"f5980794d83cf3d7807a7e9f98f8707d99209dcc4a468db8142ea4c7829d0593"} Sep 11 14:05:52.221601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:52.221562 2566 generic.go:356] "Generic (PLEG): container finished" podID="b8119fc8-fece-463f-8abb-c549cbbb0126" containerID="f5980794d83cf3d7807a7e9f98f8707d99209dcc4a468db8142ea4c7829d0593" exitCode=0 Sep 11 14:05:52.221989 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:52.221645 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" event={"ID":"b8119fc8-fece-463f-8abb-c549cbbb0126","Type":"ContainerDied","Data":"f5980794d83cf3d7807a7e9f98f8707d99209dcc4a468db8142ea4c7829d0593"} Sep 11 14:05:53.229611 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:53.229571 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" event={"ID":"b8119fc8-fece-463f-8abb-c549cbbb0126","Type":"ContainerStarted","Data":"4910a5ce188c0017c2392ac0f0b76552cb04c9d3705d34a57a41b29237e73185"} Sep 11 14:05:53.230003 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:53.229707 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:05:53.258384 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:05:53.258323 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" podStartSLOduration=2.74354215 podStartE2EDuration="7.258308024s" totalImagesPullingTime="3.438938515s" totalInitContainerRuntime="1.075827359s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:05:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:05:53.257344428 +0000 UTC m=+493.662325557" watchObservedRunningTime="2026-09-11 14:05:53.258308024 +0000 UTC m=+493.663289152" Sep 11 14:06:04.236009 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:04.235977 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5586bcdcf6-z2w7n" Sep 11 14:06:07.070152 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.070120 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8494df7896-tszz7"] Sep 11 14:06:07.070601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.070460 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c65048a-744b-454b-a887-1a39ae83a407" containerName="util" Sep 11 14:06:07.070601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.070472 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c65048a-744b-454b-a887-1a39ae83a407" containerName="util" Sep 11 14:06:07.070601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.070479 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c65048a-744b-454b-a887-1a39ae83a407" containerName="extract" Sep 11 14:06:07.070601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.070484 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c65048a-744b-454b-a887-1a39ae83a407" containerName="extract" Sep 11 14:06:07.070601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.070494 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c65048a-744b-454b-a887-1a39ae83a407" containerName="pull" Sep 11 14:06:07.070601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.070499 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c65048a-744b-454b-a887-1a39ae83a407" containerName="pull" Sep 11 14:06:07.070601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.070550 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c65048a-744b-454b-a887-1a39ae83a407" containerName="extract" Sep 11 14:06:07.072571 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.072556 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8494df7896-tszz7" Sep 11 14:06:07.091273 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.091236 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 11 14:06:07.091503 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.091473 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 14:06:07.091586 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.091568 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 14:06:07.092222 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:06:07.092203 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-tls\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" Sep 11 14:06:07.107779 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.107751 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8494df7896-tszz7"] Sep 11 14:06:07.130333 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.130299 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl94r\" (UniqueName: \"kubernetes.io/projected/45ab0a34-6a08-4cc3-9754-549294bd0b74-kube-api-access-pl94r\") pod \"kube-auth-proxy-8494df7896-tszz7\" (UID: \"45ab0a34-6a08-4cc3-9754-549294bd0b74\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-tszz7" Sep 11 14:06:07.130493 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.130374 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45ab0a34-6a08-4cc3-9754-549294bd0b74-tls-certs\") pod \"kube-auth-proxy-8494df7896-tszz7\" (UID: \"45ab0a34-6a08-4cc3-9754-549294bd0b74\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-tszz7" Sep 11 14:06:07.130493 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.130423 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45ab0a34-6a08-4cc3-9754-549294bd0b74-tmp\") pod \"kube-auth-proxy-8494df7896-tszz7\" (UID: \"45ab0a34-6a08-4cc3-9754-549294bd0b74\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-tszz7" Sep 11 14:06:07.231672 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.231641 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45ab0a34-6a08-4cc3-9754-549294bd0b74-tls-certs\") pod \"kube-auth-proxy-8494df7896-tszz7\" (UID: \"45ab0a34-6a08-4cc3-9754-549294bd0b74\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-tszz7" Sep 11 14:06:07.231860 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.231686 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45ab0a34-6a08-4cc3-9754-549294bd0b74-tmp\") pod \"kube-auth-proxy-8494df7896-tszz7\" (UID: \"45ab0a34-6a08-4cc3-9754-549294bd0b74\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-tszz7" Sep 11 14:06:07.231860 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.231721 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl94r\" (UniqueName: \"kubernetes.io/projected/45ab0a34-6a08-4cc3-9754-549294bd0b74-kube-api-access-pl94r\") pod \"kube-auth-proxy-8494df7896-tszz7\" (UID: \"45ab0a34-6a08-4cc3-9754-549294bd0b74\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-tszz7" Sep 11 14:06:07.234176 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.234144 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45ab0a34-6a08-4cc3-9754-549294bd0b74-tmp\") pod \"kube-auth-proxy-8494df7896-tszz7\" (UID: \"45ab0a34-6a08-4cc3-9754-549294bd0b74\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-tszz7" Sep 11 14:06:07.259721 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:07.259690 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl94r\" (UniqueName: \"kubernetes.io/projected/45ab0a34-6a08-4cc3-9754-549294bd0b74-kube-api-access-pl94r\") pod \"kube-auth-proxy-8494df7896-tszz7\" (UID: \"45ab0a34-6a08-4cc3-9754-549294bd0b74\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-tszz7" Sep 11 14:06:08.232488 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:06:08.232452 2566 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: failed to sync secret cache: timed out waiting for the condition Sep 11 14:06:08.232863 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:06:08.232543 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45ab0a34-6a08-4cc3-9754-549294bd0b74-tls-certs podName:45ab0a34-6a08-4cc3-9754-549294bd0b74 nodeName:}" failed. No retries permitted until 2026-09-11 14:06:08.732526172 +0000 UTC m=+509.137507278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/45ab0a34-6a08-4cc3-9754-549294bd0b74-tls-certs") pod "kube-auth-proxy-8494df7896-tszz7" (UID: "45ab0a34-6a08-4cc3-9754-549294bd0b74") : failed to sync secret cache: timed out waiting for the condition Sep 11 14:06:08.499217 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:08.499134 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 11 14:06:08.743478 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:08.743435 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45ab0a34-6a08-4cc3-9754-549294bd0b74-tls-certs\") pod \"kube-auth-proxy-8494df7896-tszz7\" (UID: \"45ab0a34-6a08-4cc3-9754-549294bd0b74\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-tszz7" Sep 11 14:06:08.746135 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:08.746078 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45ab0a34-6a08-4cc3-9754-549294bd0b74-tls-certs\") pod \"kube-auth-proxy-8494df7896-tszz7\" (UID: \"45ab0a34-6a08-4cc3-9754-549294bd0b74\") " pod="openshift-ingress/kube-auth-proxy-8494df7896-tszz7" Sep 11 14:06:08.882055 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:08.881969 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8494df7896-tszz7" Sep 11 14:06:09.197576 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.197502 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8494df7896-tszz7"] Sep 11 14:06:09.227721 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:06:09.227693 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ab0a34_6a08_4cc3_9754_549294bd0b74.slice/crio-6bb4e32e794a39f4e40c75301999c0bcd1b690fb2fb4942ba073e1c0de7086fd WatchSource:0}: Error finding container 6bb4e32e794a39f4e40c75301999c0bcd1b690fb2fb4942ba073e1c0de7086fd: Status 404 returned error can't find the container with id 6bb4e32e794a39f4e40c75301999c0bcd1b690fb2fb4942ba073e1c0de7086fd Sep 11 14:06:09.303945 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.303910 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8494df7896-tszz7" event={"ID":"45ab0a34-6a08-4cc3-9754-549294bd0b74","Type":"ContainerStarted","Data":"6bb4e32e794a39f4e40c75301999c0bcd1b690fb2fb4942ba073e1c0de7086fd"} Sep 11 14:06:09.648687 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.648640 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86"] Sep 11 14:06:09.651673 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.651653 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.654562 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.654544 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 11 14:06:09.654687 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.654546 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 11 14:06:09.654859 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.654845 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fbfbr\"" Sep 11 14:06:09.655684 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.655667 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 14:06:09.655768 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.655700 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 11 14:06:09.677674 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.677640 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86"] Sep 11 14:06:09.751502 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.751464 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.751679 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.751546 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.751679 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.751582 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d59a058-c231-481f-a579-95af087b9050-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.751679 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.751669 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0d59a058-c231-481f-a579-95af087b9050-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.751840 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.751695 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggqtr\" (UniqueName: \"kubernetes.io/projected/0d59a058-c231-481f-a579-95af087b9050-kube-api-access-ggqtr\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.751840 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.751719 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.751840 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.751789 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d59a058-c231-481f-a579-95af087b9050-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.852856 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.852818 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.852856 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.852862 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d59a058-c231-481f-a579-95af087b9050-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.853144 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.852961 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0d59a058-c231-481f-a579-95af087b9050-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.853144 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.852987 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqtr\" (UniqueName: \"kubernetes.io/projected/0d59a058-c231-481f-a579-95af087b9050-kube-api-access-ggqtr\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.853144 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.853013 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.853144 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.853066 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d59a058-c231-481f-a579-95af087b9050-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.853144 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.853132 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.853843 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.853796 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0d59a058-c231-481f-a579-95af087b9050-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.856532 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.856485 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.856532 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.856523 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.856789 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.856764 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d59a058-c231-481f-a579-95af087b9050-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.856907 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.856804 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.868666 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.868620 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqtr\" (UniqueName: \"kubernetes.io/projected/0d59a058-c231-481f-a579-95af087b9050-kube-api-access-ggqtr\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.869988 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.869953 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d59a058-c231-481f-a579-95af087b9050-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-44l86\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:09.961314 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:09.961233 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:10.284484 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:10.284455 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86"] Sep 11 14:06:10.288025 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:06:10.287995 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d59a058_c231_481f_a579_95af087b9050.slice/crio-33a40845244b1f68e6980b302191cdcf58ec879cf0de03a99a6bfb2c6fed13e3 WatchSource:0}: Error finding container 33a40845244b1f68e6980b302191cdcf58ec879cf0de03a99a6bfb2c6fed13e3: Status 404 returned error can't find the container with id 33a40845244b1f68e6980b302191cdcf58ec879cf0de03a99a6bfb2c6fed13e3 Sep 11 14:06:10.312121 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:10.312059 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" event={"ID":"0d59a058-c231-481f-a579-95af087b9050","Type":"ContainerStarted","Data":"33a40845244b1f68e6980b302191cdcf58ec879cf0de03a99a6bfb2c6fed13e3"} Sep 11 14:06:12.326988 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:12.326954 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8494df7896-tszz7" event={"ID":"45ab0a34-6a08-4cc3-9754-549294bd0b74","Type":"ContainerStarted","Data":"5bb36a9a9d0dd652cbb306dc941b25a19946d827fef3e89c85074c4d4f2b971a"} Sep 11 14:06:12.383945 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:12.383880 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8494df7896-tszz7" podStartSLOduration=2.8535872209999997 podStartE2EDuration="5.383856145s" totalImagesPullingTime="2.530268924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:12.383399312 +0000 UTC m=+512.788380441" watchObservedRunningTime="2026-09-11 14:06:12.383856145 +0000 UTC m=+512.788837275" Sep 11 14:06:13.526503 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:13.526470 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:13.526734 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:13.526540 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:14.338670 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:14.338634 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" event={"ID":"0d59a058-c231-481f-a579-95af087b9050","Type":"ContainerStarted","Data":"9b84ed29c470dd281a182d55f1b34c45eb6c31d61fea38aa6a83d1e9f185c347"} Sep 11 14:06:14.338882 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:14.338697 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:14.340471 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:14.340445 2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-44l86 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:06:14.340576 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:14.340497 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" podUID="0d59a058-c231-481f-a579-95af087b9050" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:06:14.364530 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:14.364484 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" podStartSLOduration=2.128893007 podStartE2EDuration="5.364470459s" totalImagesPullingTime="3.235577452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:14.363416613 +0000 UTC m=+514.768397766" watchObservedRunningTime="2026-09-11 14:06:14.364470459 +0000 UTC m=+514.769451586" Sep 11 14:06:15.343705 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.343666 2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-44l86 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:06:15.344077 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.343720 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" podUID="0d59a058-c231-481f-a579-95af087b9050" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:06:15.714963 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.714890 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht"] Sep 11 14:06:15.717326 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.717304 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.721190 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.721165 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5xqcx\"" Sep 11 14:06:15.746375 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.746343 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht"] Sep 11 14:06:15.808363 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.808326 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f08071d-1a99-4cd9-8929-f202cd23051d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.808363 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.808365 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f08071d-1a99-4cd9-8929-f202cd23051d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.808600 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.808391 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f08071d-1a99-4cd9-8929-f202cd23051d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.808600 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.808518 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f08071d-1a99-4cd9-8929-f202cd23051d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.808600 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.808560 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f08071d-1a99-4cd9-8929-f202cd23051d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.808754 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.808607 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f08071d-1a99-4cd9-8929-f202cd23051d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.808754 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.808666 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f08071d-1a99-4cd9-8929-f202cd23051d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.808754 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.808713 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt97k\" (UniqueName: \"kubernetes.io/projected/4f08071d-1a99-4cd9-8929-f202cd23051d-kube-api-access-zt97k\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.808858 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.808759 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f08071d-1a99-4cd9-8929-f202cd23051d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.910152 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.910075 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f08071d-1a99-4cd9-8929-f202cd23051d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.910325 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.910169 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f08071d-1a99-4cd9-8929-f202cd23051d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.910325 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.910220 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f08071d-1a99-4cd9-8929-f202cd23051d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.910325 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.910278 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f08071d-1a99-4cd9-8929-f202cd23051d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.910325 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.910315 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f08071d-1a99-4cd9-8929-f202cd23051d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.910694 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.910665 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f08071d-1a99-4cd9-8929-f202cd23051d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.910782 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.910724 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f08071d-1a99-4cd9-8929-f202cd23051d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.910833 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.910799 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f08071d-1a99-4cd9-8929-f202cd23051d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.910903 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.910857 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f08071d-1a99-4cd9-8929-f202cd23051d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.910959 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.910913 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt97k\" (UniqueName: \"kubernetes.io/projected/4f08071d-1a99-4cd9-8929-f202cd23051d-kube-api-access-zt97k\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.910959 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.910929 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f08071d-1a99-4cd9-8929-f202cd23051d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.910959 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.910944 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f08071d-1a99-4cd9-8929-f202cd23051d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.911268 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.911169 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f08071d-1a99-4cd9-8929-f202cd23051d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.911268 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.911276 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f08071d-1a99-4cd9-8929-f202cd23051d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.913242 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.913219 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f08071d-1a99-4cd9-8929-f202cd23051d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.913339 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.913220 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f08071d-1a99-4cd9-8929-f202cd23051d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.921177 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.921155 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt97k\" (UniqueName: \"kubernetes.io/projected/4f08071d-1a99-4cd9-8929-f202cd23051d-kube-api-access-zt97k\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:15.922897 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:15.922877 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f08071d-1a99-4cd9-8929-f202cd23051d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht\" (UID: \"4f08071d-1a99-4cd9-8929-f202cd23051d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:16.028276 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:16.028236 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:16.329521 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:06:16.329490 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f08071d_1a99_4cd9_8929_f202cd23051d.slice/crio-74e8baa247ef7c85acf04e2a1d7cc521d73ea9d0da968d75699c5f4f24023879 WatchSource:0}: Error finding container 74e8baa247ef7c85acf04e2a1d7cc521d73ea9d0da968d75699c5f4f24023879: Status 404 returned error can't find the container with id 74e8baa247ef7c85acf04e2a1d7cc521d73ea9d0da968d75699c5f4f24023879 Sep 11 14:06:16.338161 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:16.338136 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht"] Sep 11 14:06:16.350444 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:16.350398 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" event={"ID":"4f08071d-1a99-4cd9-8929-f202cd23051d","Type":"ContainerStarted","Data":"74e8baa247ef7c85acf04e2a1d7cc521d73ea9d0da968d75699c5f4f24023879"} Sep 11 14:06:18.344343 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:18.344306 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:19.591266 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:19.591236 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:19.591507 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:19.591305 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:19.591507 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:19.591336 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:20.372477 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:20.372440 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" event={"ID":"4f08071d-1a99-4cd9-8929-f202cd23051d","Type":"ContainerStarted","Data":"43411a87e9cd77281a6445fc9dd8af5d2bdf5afa4fa92f6b61b4fbe22132b292"} Sep 11 14:06:21.029158 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:21.029112 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:21.035382 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:21.035357 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:21.058360 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:21.058303 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" podStartSLOduration=2.798754827 podStartE2EDuration="6.058287285s" totalImagesPullingTime="3.259532458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:20.400214861 +0000 UTC m=+520.805195989" watchObservedRunningTime="2026-09-11 14:06:21.058287285 +0000 UTC m=+521.463268413" Sep 11 14:06:21.378168 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:21.378049 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:21.379196 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:21.379171 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht" Sep 11 14:06:29.136405 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.136373 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs"] Sep 11 14:06:29.144244 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.144220 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs" Sep 11 14:06:29.153434 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.153412 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 11 14:06:29.153561 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.153494 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-x4blc\"" Sep 11 14:06:29.171482 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.171454 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs"] Sep 11 14:06:29.193854 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.193818 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb"] Sep 11 14:06:29.196063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.196046 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb" Sep 11 14:06:29.213948 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.213924 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-sggx4\"" Sep 11 14:06:29.237188 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.237164 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b4bed51-642c-46d6-a9f7-a1184b6d21be-cert\") pod \"kserve-controller-manager-d6f4b65b8-jxzrs\" (UID: \"2b4bed51-642c-46d6-a9f7-a1184b6d21be\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs" Sep 11 14:06:29.237321 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.237227 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbgdt\" (UniqueName: \"kubernetes.io/projected/2b4bed51-642c-46d6-a9f7-a1184b6d21be-kube-api-access-vbgdt\") pod \"kserve-controller-manager-d6f4b65b8-jxzrs\" (UID: \"2b4bed51-642c-46d6-a9f7-a1184b6d21be\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs" Sep 11 14:06:29.238621 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.238598 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 11 14:06:29.259191 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.259161 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb"] Sep 11 14:06:29.337726 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.337687 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/922f7c36-5337-45e2-a7b6-94f1ced90673-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vwzkb\" (UID: \"922f7c36-5337-45e2-a7b6-94f1ced90673\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb" Sep 11 14:06:29.337909 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.337729 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbgdt\" (UniqueName: \"kubernetes.io/projected/2b4bed51-642c-46d6-a9f7-a1184b6d21be-kube-api-access-vbgdt\") pod \"kserve-controller-manager-d6f4b65b8-jxzrs\" (UID: \"2b4bed51-642c-46d6-a9f7-a1184b6d21be\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs" Sep 11 14:06:29.337909 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.337786 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5vrb\" (UniqueName: \"kubernetes.io/projected/922f7c36-5337-45e2-a7b6-94f1ced90673-kube-api-access-z5vrb\") pod \"llmisvc-controller-manager-777c6bd7d5-vwzkb\" (UID: \"922f7c36-5337-45e2-a7b6-94f1ced90673\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb" Sep 11 14:06:29.337909 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.337847 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b4bed51-642c-46d6-a9f7-a1184b6d21be-cert\") pod \"kserve-controller-manager-d6f4b65b8-jxzrs\" (UID: \"2b4bed51-642c-46d6-a9f7-a1184b6d21be\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs" Sep 11 14:06:29.340539 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.340513 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b4bed51-642c-46d6-a9f7-a1184b6d21be-cert\") pod \"kserve-controller-manager-d6f4b65b8-jxzrs\" (UID: \"2b4bed51-642c-46d6-a9f7-a1184b6d21be\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs" Sep 11 14:06:29.376807 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.376769 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbgdt\" (UniqueName: \"kubernetes.io/projected/2b4bed51-642c-46d6-a9f7-a1184b6d21be-kube-api-access-vbgdt\") pod \"kserve-controller-manager-d6f4b65b8-jxzrs\" (UID: \"2b4bed51-642c-46d6-a9f7-a1184b6d21be\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs" Sep 11 14:06:29.438663 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.438574 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/922f7c36-5337-45e2-a7b6-94f1ced90673-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vwzkb\" (UID: \"922f7c36-5337-45e2-a7b6-94f1ced90673\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb" Sep 11 14:06:29.438663 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.438625 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5vrb\" (UniqueName: \"kubernetes.io/projected/922f7c36-5337-45e2-a7b6-94f1ced90673-kube-api-access-z5vrb\") pod \"llmisvc-controller-manager-777c6bd7d5-vwzkb\" (UID: \"922f7c36-5337-45e2-a7b6-94f1ced90673\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb" Sep 11 14:06:29.438860 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:06:29.438729 2566 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 11 14:06:29.438860 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:06:29.438791 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/922f7c36-5337-45e2-a7b6-94f1ced90673-cert podName:922f7c36-5337-45e2-a7b6-94f1ced90673 nodeName:}" failed. No retries permitted until 2026-09-11 14:06:29.938773609 +0000 UTC m=+530.343754718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/922f7c36-5337-45e2-a7b6-94f1ced90673-cert") pod "llmisvc-controller-manager-777c6bd7d5-vwzkb" (UID: "922f7c36-5337-45e2-a7b6-94f1ced90673") : secret "llmisvc-webhook-server-cert" not found Sep 11 14:06:29.453638 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.453610 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs" Sep 11 14:06:29.462835 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.462807 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5vrb\" (UniqueName: \"kubernetes.io/projected/922f7c36-5337-45e2-a7b6-94f1ced90673-kube-api-access-z5vrb\") pod \"llmisvc-controller-manager-777c6bd7d5-vwzkb\" (UID: \"922f7c36-5337-45e2-a7b6-94f1ced90673\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb" Sep 11 14:06:29.764538 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.764510 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs"] Sep 11 14:06:29.767271 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:06:29.767244 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4bed51_642c_46d6_a9f7_a1184b6d21be.slice/crio-24020a560f4d7bc04dfe86816f7012660dc683852b7c156f33f17bca24bd0bf8 WatchSource:0}: Error finding container 24020a560f4d7bc04dfe86816f7012660dc683852b7c156f33f17bca24bd0bf8: Status 404 returned error can't find the container with id 24020a560f4d7bc04dfe86816f7012660dc683852b7c156f33f17bca24bd0bf8 Sep 11 14:06:29.943281 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.943242 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/922f7c36-5337-45e2-a7b6-94f1ced90673-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vwzkb\" (UID: \"922f7c36-5337-45e2-a7b6-94f1ced90673\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb" Sep 11 14:06:29.946062 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:29.946028 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/922f7c36-5337-45e2-a7b6-94f1ced90673-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vwzkb\" (UID: \"922f7c36-5337-45e2-a7b6-94f1ced90673\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb" Sep 11 14:06:30.105158 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:30.105048 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb" Sep 11 14:06:30.421766 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:30.421677 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs" event={"ID":"2b4bed51-642c-46d6-a9f7-a1184b6d21be","Type":"ContainerStarted","Data":"24020a560f4d7bc04dfe86816f7012660dc683852b7c156f33f17bca24bd0bf8"} Sep 11 14:06:30.517288 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:06:30.517255 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod922f7c36_5337_45e2_a7b6_94f1ced90673.slice/crio-756986ab6f178dbcb41dab0e640a2a3ea3f8231466aa9acd3528569437acc7fb WatchSource:0}: Error finding container 756986ab6f178dbcb41dab0e640a2a3ea3f8231466aa9acd3528569437acc7fb: Status 404 returned error can't find the container with id 756986ab6f178dbcb41dab0e640a2a3ea3f8231466aa9acd3528569437acc7fb Sep 11 14:06:30.528926 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:30.528897 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb"] Sep 11 14:06:30.999827 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:30.999790 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86"] Sep 11 14:06:31.000052 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:31.000021 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" podUID="0d59a058-c231-481f-a579-95af087b9050" containerName="discovery" containerID="cri-o://9b84ed29c470dd281a182d55f1b34c45eb6c31d61fea38aa6a83d1e9f185c347" gracePeriod=30 Sep 11 14:06:31.428284 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:31.428196 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb" event={"ID":"922f7c36-5337-45e2-a7b6-94f1ced90673","Type":"ContainerStarted","Data":"756986ab6f178dbcb41dab0e640a2a3ea3f8231466aa9acd3528569437acc7fb"} Sep 11 14:06:31.430002 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:31.429969 2566 generic.go:356] "Generic (PLEG): container finished" podID="0d59a058-c231-481f-a579-95af087b9050" containerID="9b84ed29c470dd281a182d55f1b34c45eb6c31d61fea38aa6a83d1e9f185c347" exitCode=0 Sep 11 14:06:31.430164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:31.430055 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" event={"ID":"0d59a058-c231-481f-a579-95af087b9050","Type":"ContainerDied","Data":"9b84ed29c470dd281a182d55f1b34c45eb6c31d61fea38aa6a83d1e9f185c347"} Sep 11 14:06:31.977176 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:31.977151 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:32.064564 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.064517 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggqtr\" (UniqueName: \"kubernetes.io/projected/0d59a058-c231-481f-a579-95af087b9050-kube-api-access-ggqtr\") pod \"0d59a058-c231-481f-a579-95af087b9050\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " Sep 11 14:06:32.064564 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.064566 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d59a058-c231-481f-a579-95af087b9050-local-certs\") pod \"0d59a058-c231-481f-a579-95af087b9050\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " Sep 11 14:06:32.064817 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.064598 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-istio-csr-dns-cert\") pod \"0d59a058-c231-481f-a579-95af087b9050\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " Sep 11 14:06:32.064817 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.064641 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-cacerts\") pod \"0d59a058-c231-481f-a579-95af087b9050\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " Sep 11 14:06:32.064817 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.064671 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-istio-kubeconfig\") pod \"0d59a058-c231-481f-a579-95af087b9050\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " Sep 11 14:06:32.064817 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.064700 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0d59a058-c231-481f-a579-95af087b9050-istio-csr-ca-configmap\") pod \"0d59a058-c231-481f-a579-95af087b9050\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " Sep 11 14:06:32.064817 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.064760 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d59a058-c231-481f-a579-95af087b9050-istio-token\") pod \"0d59a058-c231-481f-a579-95af087b9050\" (UID: \"0d59a058-c231-481f-a579-95af087b9050\") " Sep 11 14:06:32.065789 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.065721 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d59a058-c231-481f-a579-95af087b9050-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "0d59a058-c231-481f-a579-95af087b9050" (UID: "0d59a058-c231-481f-a579-95af087b9050"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:06:32.067548 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.067500 2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0d59a058-c231-481f-a579-95af087b9050-istio-csr-ca-configmap\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:06:32.067961 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.067802 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-cacerts" (OuterVolumeSpecName: "cacerts") pod "0d59a058-c231-481f-a579-95af087b9050" (UID: "0d59a058-c231-481f-a579-95af087b9050"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:06:32.067961 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.067929 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d59a058-c231-481f-a579-95af087b9050-kube-api-access-ggqtr" (OuterVolumeSpecName: "kube-api-access-ggqtr") pod "0d59a058-c231-481f-a579-95af087b9050" (UID: "0d59a058-c231-481f-a579-95af087b9050"). InnerVolumeSpecName "kube-api-access-ggqtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:06:32.068478 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.068455 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "0d59a058-c231-481f-a579-95af087b9050" (UID: "0d59a058-c231-481f-a579-95af087b9050"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:06:32.069325 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.069288 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d59a058-c231-481f-a579-95af087b9050-local-certs" (OuterVolumeSpecName: "local-certs") pod "0d59a058-c231-481f-a579-95af087b9050" (UID: "0d59a058-c231-481f-a579-95af087b9050"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:06:32.069415 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.069330 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d59a058-c231-481f-a579-95af087b9050-istio-token" (OuterVolumeSpecName: "istio-token") pod "0d59a058-c231-481f-a579-95af087b9050" (UID: "0d59a058-c231-481f-a579-95af087b9050"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:06:32.069818 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.069800 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "0d59a058-c231-481f-a579-95af087b9050" (UID: "0d59a058-c231-481f-a579-95af087b9050"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:06:32.168761 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.168621 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d59a058-c231-481f-a579-95af087b9050-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:06:32.168761 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.168659 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggqtr\" (UniqueName: \"kubernetes.io/projected/0d59a058-c231-481f-a579-95af087b9050-kube-api-access-ggqtr\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:06:32.168761 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.168675 2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d59a058-c231-481f-a579-95af087b9050-local-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:06:32.168761 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.168695 2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-istio-csr-dns-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:06:32.168761 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.168719 2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-cacerts\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:06:32.168761 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.168733 2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d59a058-c231-481f-a579-95af087b9050-istio-kubeconfig\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:06:32.438053 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.437988 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" Sep 11 14:06:32.438535 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.437990 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86" event={"ID":"0d59a058-c231-481f-a579-95af087b9050","Type":"ContainerDied","Data":"33a40845244b1f68e6980b302191cdcf58ec879cf0de03a99a6bfb2c6fed13e3"} Sep 11 14:06:32.438535 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.438162 2566 scope.go:117] "RemoveContainer" containerID="9b84ed29c470dd281a182d55f1b34c45eb6c31d61fea38aa6a83d1e9f185c347" Sep 11 14:06:32.440325 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.440266 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs" event={"ID":"2b4bed51-642c-46d6-a9f7-a1184b6d21be","Type":"ContainerStarted","Data":"8b615fbce58a03c596ebf10bd8c8335df7f14f3365e85d671d012affa7a3dd6a"} Sep 11 14:06:32.440647 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:32.440613 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs" Sep 11 14:06:34.453593 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:34.453556 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb" event={"ID":"922f7c36-5337-45e2-a7b6-94f1ced90673","Type":"ContainerStarted","Data":"141300d5102eb8a533f5e4544276443c6d9124e4df91dfbe43213b665ab908a4"} Sep 11 14:06:34.454002 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:34.453776 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb" Sep 11 14:06:35.590374 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:35.590343 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86"] Sep 11 14:06:37.603620 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:37.603585 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-44l86"] Sep 11 14:06:38.097777 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:38.097742 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d59a058-c231-481f-a579-95af087b9050" path="/var/lib/kubelet/pods/0d59a058-c231-481f-a579-95af087b9050/volumes" Sep 11 14:06:41.039291 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:41.039229 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs" podStartSLOduration=9.689583914 podStartE2EDuration="12.039211543s" totalImagesPullingTime="2.349627629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:41.037903211 +0000 UTC m=+541.442884338" watchObservedRunningTime="2026-09-11 14:06:41.039211543 +0000 UTC m=+541.444192671" Sep 11 14:06:41.199395 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:06:41.199345 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb" podStartSLOduration=8.935574527 podStartE2EDuration="12.199328765s" totalImagesPullingTime="3.263754238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:41.198494621 +0000 UTC m=+541.603475748" watchObservedRunningTime="2026-09-11 14:06:41.199328765 +0000 UTC m=+541.604309922" Sep 11 14:07:04.459458 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:04.459427 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jxzrs" Sep 11 14:07:05.460896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:05.460859 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vwzkb" Sep 11 14:07:24.293349 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.293318 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn"] Sep 11 14:07:24.293750 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.293635 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d59a058-c231-481f-a579-95af087b9050" containerName="discovery" Sep 11 14:07:24.293750 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.293646 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d59a058-c231-481f-a579-95af087b9050" containerName="discovery" Sep 11 14:07:24.293750 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.293711 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d59a058-c231-481f-a579-95af087b9050" containerName="discovery" Sep 11 14:07:24.298562 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.298546 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" Sep 11 14:07:24.301615 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.301588 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-w8ql7\"" Sep 11 14:07:24.301864 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.301846 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 14:07:24.302338 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.302322 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 14:07:24.306346 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.306323 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn"] Sep 11 14:07:24.394051 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.394015 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c"] Sep 11 14:07:24.396355 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.396336 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" Sep 11 14:07:24.407997 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.407971 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c"] Sep 11 14:07:24.408283 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.408260 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgccn\" (UniqueName: \"kubernetes.io/projected/0903f6d9-52a5-424c-954d-5296905d47d1-kube-api-access-pgccn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn\" (UID: \"0903f6d9-52a5-424c-954d-5296905d47d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" Sep 11 14:07:24.408348 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.408305 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0903f6d9-52a5-424c-954d-5296905d47d1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn\" (UID: \"0903f6d9-52a5-424c-954d-5296905d47d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" Sep 11 14:07:24.408348 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.408330 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0903f6d9-52a5-424c-954d-5296905d47d1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn\" (UID: \"0903f6d9-52a5-424c-954d-5296905d47d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" Sep 11 14:07:24.495208 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.495171 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r"] Sep 11 14:07:24.497622 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.497605 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" Sep 11 14:07:24.509147 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.509118 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r"] Sep 11 14:07:24.510031 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.510006 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgccn\" (UniqueName: \"kubernetes.io/projected/0903f6d9-52a5-424c-954d-5296905d47d1-kube-api-access-pgccn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn\" (UID: \"0903f6d9-52a5-424c-954d-5296905d47d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" Sep 11 14:07:24.510238 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.510219 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4875ca2f-287a-4774-926f-84326befcabf-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c\" (UID: \"4875ca2f-287a-4774-926f-84326befcabf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" Sep 11 14:07:24.510387 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.510372 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kg25\" (UniqueName: \"kubernetes.io/projected/4875ca2f-287a-4774-926f-84326befcabf-kube-api-access-9kg25\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c\" (UID: \"4875ca2f-287a-4774-926f-84326befcabf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" Sep 11 14:07:24.510526 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.510514 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0903f6d9-52a5-424c-954d-5296905d47d1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn\" (UID: \"0903f6d9-52a5-424c-954d-5296905d47d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" Sep 11 14:07:24.510979 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.510963 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0903f6d9-52a5-424c-954d-5296905d47d1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn\" (UID: \"0903f6d9-52a5-424c-954d-5296905d47d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" Sep 11 14:07:24.511461 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.511446 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4875ca2f-287a-4774-926f-84326befcabf-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c\" (UID: \"4875ca2f-287a-4774-926f-84326befcabf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" Sep 11 14:07:24.511635 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.510912 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0903f6d9-52a5-424c-954d-5296905d47d1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn\" (UID: \"0903f6d9-52a5-424c-954d-5296905d47d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" Sep 11 14:07:24.511724 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.511323 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0903f6d9-52a5-424c-954d-5296905d47d1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn\" (UID: \"0903f6d9-52a5-424c-954d-5296905d47d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" Sep 11 14:07:24.527851 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.527822 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgccn\" (UniqueName: \"kubernetes.io/projected/0903f6d9-52a5-424c-954d-5296905d47d1-kube-api-access-pgccn\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn\" (UID: \"0903f6d9-52a5-424c-954d-5296905d47d1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" Sep 11 14:07:24.597021 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.596940 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh"] Sep 11 14:07:24.599377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.599362 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" Sep 11 14:07:24.608425 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.608402 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" Sep 11 14:07:24.610818 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.610800 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh"] Sep 11 14:07:24.612609 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.612585 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff7145cb-d315-436a-867d-f62cb65214d0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r\" (UID: \"ff7145cb-d315-436a-867d-f62cb65214d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" Sep 11 14:07:24.612709 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.612623 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xftsv\" (UniqueName: \"kubernetes.io/projected/ff7145cb-d315-436a-867d-f62cb65214d0-kube-api-access-xftsv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r\" (UID: \"ff7145cb-d315-436a-867d-f62cb65214d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" Sep 11 14:07:24.612709 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.612656 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4875ca2f-287a-4774-926f-84326befcabf-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c\" (UID: \"4875ca2f-287a-4774-926f-84326befcabf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" Sep 11 14:07:24.612791 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.612736 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4875ca2f-287a-4774-926f-84326befcabf-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c\" (UID: \"4875ca2f-287a-4774-926f-84326befcabf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" Sep 11 14:07:24.612791 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.612777 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kg25\" (UniqueName: \"kubernetes.io/projected/4875ca2f-287a-4774-926f-84326befcabf-kube-api-access-9kg25\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c\" (UID: \"4875ca2f-287a-4774-926f-84326befcabf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" Sep 11 14:07:24.612904 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.612851 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff7145cb-d315-436a-867d-f62cb65214d0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r\" (UID: \"ff7145cb-d315-436a-867d-f62cb65214d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" Sep 11 14:07:24.613021 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.612998 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4875ca2f-287a-4774-926f-84326befcabf-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c\" (UID: \"4875ca2f-287a-4774-926f-84326befcabf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" Sep 11 14:07:24.613133 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.613116 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4875ca2f-287a-4774-926f-84326befcabf-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c\" (UID: \"4875ca2f-287a-4774-926f-84326befcabf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" Sep 11 14:07:24.623962 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.623940 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kg25\" (UniqueName: \"kubernetes.io/projected/4875ca2f-287a-4774-926f-84326befcabf-kube-api-access-9kg25\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c\" (UID: \"4875ca2f-287a-4774-926f-84326befcabf\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" Sep 11 14:07:24.705504 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.705473 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" Sep 11 14:07:24.713647 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.713620 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgm25\" (UniqueName: \"kubernetes.io/projected/7fc465ec-a28b-4185-ae4d-19e134e15d48-kube-api-access-hgm25\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh\" (UID: \"7fc465ec-a28b-4185-ae4d-19e134e15d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" Sep 11 14:07:24.713792 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.713658 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fc465ec-a28b-4185-ae4d-19e134e15d48-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh\" (UID: \"7fc465ec-a28b-4185-ae4d-19e134e15d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" Sep 11 14:07:24.713792 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.713688 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff7145cb-d315-436a-867d-f62cb65214d0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r\" (UID: \"ff7145cb-d315-436a-867d-f62cb65214d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" Sep 11 14:07:24.713792 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.713780 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff7145cb-d315-436a-867d-f62cb65214d0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r\" (UID: \"ff7145cb-d315-436a-867d-f62cb65214d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" Sep 11 14:07:24.713952 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.713820 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fc465ec-a28b-4185-ae4d-19e134e15d48-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh\" (UID: \"7fc465ec-a28b-4185-ae4d-19e134e15d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" Sep 11 14:07:24.713952 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.713938 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xftsv\" (UniqueName: \"kubernetes.io/projected/ff7145cb-d315-436a-867d-f62cb65214d0-kube-api-access-xftsv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r\" (UID: \"ff7145cb-d315-436a-867d-f62cb65214d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" Sep 11 14:07:24.714075 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.714059 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff7145cb-d315-436a-867d-f62cb65214d0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r\" (UID: \"ff7145cb-d315-436a-867d-f62cb65214d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" Sep 11 14:07:24.714166 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.714138 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff7145cb-d315-436a-867d-f62cb65214d0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r\" (UID: \"ff7145cb-d315-436a-867d-f62cb65214d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" Sep 11 14:07:24.724152 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.724121 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xftsv\" (UniqueName: \"kubernetes.io/projected/ff7145cb-d315-436a-867d-f62cb65214d0-kube-api-access-xftsv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r\" (UID: \"ff7145cb-d315-436a-867d-f62cb65214d0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" Sep 11 14:07:24.806705 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.806676 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" Sep 11 14:07:24.815355 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.815324 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgm25\" (UniqueName: \"kubernetes.io/projected/7fc465ec-a28b-4185-ae4d-19e134e15d48-kube-api-access-hgm25\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh\" (UID: \"7fc465ec-a28b-4185-ae4d-19e134e15d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" Sep 11 14:07:24.815455 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.815376 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fc465ec-a28b-4185-ae4d-19e134e15d48-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh\" (UID: \"7fc465ec-a28b-4185-ae4d-19e134e15d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" Sep 11 14:07:24.815455 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.815435 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fc465ec-a28b-4185-ae4d-19e134e15d48-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh\" (UID: \"7fc465ec-a28b-4185-ae4d-19e134e15d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" Sep 11 14:07:24.815772 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.815751 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fc465ec-a28b-4185-ae4d-19e134e15d48-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh\" (UID: \"7fc465ec-a28b-4185-ae4d-19e134e15d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" Sep 11 14:07:24.815772 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.815764 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fc465ec-a28b-4185-ae4d-19e134e15d48-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh\" (UID: \"7fc465ec-a28b-4185-ae4d-19e134e15d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" Sep 11 14:07:24.826266 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.826238 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgm25\" (UniqueName: \"kubernetes.io/projected/7fc465ec-a28b-4185-ae4d-19e134e15d48-kube-api-access-hgm25\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh\" (UID: \"7fc465ec-a28b-4185-ae4d-19e134e15d48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" Sep 11 14:07:24.886675 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.886646 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn"] Sep 11 14:07:24.888505 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:07:24.888476 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0903f6d9_52a5_424c_954d_5296905d47d1.slice/crio-046609d3eef743c3333544bc98b88e6c431bbde59df59049dbf0d0865e55c688 WatchSource:0}: Error finding container 046609d3eef743c3333544bc98b88e6c431bbde59df59049dbf0d0865e55c688: Status 404 returned error can't find the container with id 046609d3eef743c3333544bc98b88e6c431bbde59df59049dbf0d0865e55c688 Sep 11 14:07:24.908219 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.908199 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" Sep 11 14:07:24.984122 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:24.984078 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c"] Sep 11 14:07:24.987194 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:07:24.987163 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4875ca2f_287a_4774_926f_84326befcabf.slice/crio-2cab24ed4b6745ef2b71f8cc332a614cd4f8942f36205eb913bdc2ce0174320c WatchSource:0}: Error finding container 2cab24ed4b6745ef2b71f8cc332a614cd4f8942f36205eb913bdc2ce0174320c: Status 404 returned error can't find the container with id 2cab24ed4b6745ef2b71f8cc332a614cd4f8942f36205eb913bdc2ce0174320c Sep 11 14:07:25.078552 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:25.078495 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r"] Sep 11 14:07:25.080289 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:07:25.080262 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7145cb_d315_436a_867d_f62cb65214d0.slice/crio-e3671dde3a4470e8c4c5313ede3b451ca8d8e4f0e1ec6ea360652fae8c7d14b4 WatchSource:0}: Error finding container e3671dde3a4470e8c4c5313ede3b451ca8d8e4f0e1ec6ea360652fae8c7d14b4: Status 404 returned error can't find the container with id e3671dde3a4470e8c4c5313ede3b451ca8d8e4f0e1ec6ea360652fae8c7d14b4 Sep 11 14:07:25.191166 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:25.191133 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh"] Sep 11 14:07:25.243183 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:07:25.243149 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc465ec_a28b_4185_ae4d_19e134e15d48.slice/crio-a7927a969dabc810310411100b08a7fd9c0ae7170165786d025890eec52825bc WatchSource:0}: Error finding container a7927a969dabc810310411100b08a7fd9c0ae7170165786d025890eec52825bc: Status 404 returned error can't find the container with id a7927a969dabc810310411100b08a7fd9c0ae7170165786d025890eec52825bc Sep 11 14:07:25.697876 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:25.697846 2566 generic.go:356] "Generic (PLEG): container finished" podID="4875ca2f-287a-4774-926f-84326befcabf" containerID="5b607d695a67318f009960651190efddf264b5539653b1bb60b36b93b41557d1" exitCode=0 Sep 11 14:07:25.698303 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:25.697939 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" event={"ID":"4875ca2f-287a-4774-926f-84326befcabf","Type":"ContainerDied","Data":"5b607d695a67318f009960651190efddf264b5539653b1bb60b36b93b41557d1"} Sep 11 14:07:25.698303 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:25.697989 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" event={"ID":"4875ca2f-287a-4774-926f-84326befcabf","Type":"ContainerStarted","Data":"2cab24ed4b6745ef2b71f8cc332a614cd4f8942f36205eb913bdc2ce0174320c"} Sep 11 14:07:25.699694 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:25.699666 2566 generic.go:356] "Generic (PLEG): container finished" podID="ff7145cb-d315-436a-867d-f62cb65214d0" containerID="a581967f18ab047ef4346b27a2d55d39816a39f491ed4d9517f31a26c81defe4" exitCode=0 Sep 11 14:07:25.699801 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:25.699751 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" event={"ID":"ff7145cb-d315-436a-867d-f62cb65214d0","Type":"ContainerDied","Data":"a581967f18ab047ef4346b27a2d55d39816a39f491ed4d9517f31a26c81defe4"} Sep 11 14:07:25.699862 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:25.699801 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" event={"ID":"ff7145cb-d315-436a-867d-f62cb65214d0","Type":"ContainerStarted","Data":"e3671dde3a4470e8c4c5313ede3b451ca8d8e4f0e1ec6ea360652fae8c7d14b4"} Sep 11 14:07:25.701499 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:25.701477 2566 generic.go:356] "Generic (PLEG): container finished" podID="0903f6d9-52a5-424c-954d-5296905d47d1" containerID="a1547c5aaf94ecb694dbb4cb6a33b510ee6d3a0bf8d6352f6ace0562d381e3e6" exitCode=0 Sep 11 14:07:25.701591 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:25.701560 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" event={"ID":"0903f6d9-52a5-424c-954d-5296905d47d1","Type":"ContainerDied","Data":"a1547c5aaf94ecb694dbb4cb6a33b510ee6d3a0bf8d6352f6ace0562d381e3e6"} Sep 11 14:07:25.701656 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:25.701598 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" event={"ID":"0903f6d9-52a5-424c-954d-5296905d47d1","Type":"ContainerStarted","Data":"046609d3eef743c3333544bc98b88e6c431bbde59df59049dbf0d0865e55c688"} Sep 11 14:07:25.703438 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:25.703416 2566 generic.go:356] "Generic (PLEG): container finished" podID="7fc465ec-a28b-4185-ae4d-19e134e15d48" containerID="86350ba1629bb522d53db8dcd3b600c6d5952bf5c2aab017d824440937dd7a85" exitCode=0 Sep 11 14:07:25.703518 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:25.703452 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" event={"ID":"7fc465ec-a28b-4185-ae4d-19e134e15d48","Type":"ContainerDied","Data":"86350ba1629bb522d53db8dcd3b600c6d5952bf5c2aab017d824440937dd7a85"} Sep 11 14:07:25.703518 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:25.703496 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" event={"ID":"7fc465ec-a28b-4185-ae4d-19e134e15d48","Type":"ContainerStarted","Data":"a7927a969dabc810310411100b08a7fd9c0ae7170165786d025890eec52825bc"} Sep 11 14:07:26.711206 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:26.711183 2566 generic.go:356] "Generic (PLEG): container finished" podID="ff7145cb-d315-436a-867d-f62cb65214d0" containerID="08a56eb0d1985ccd990490d6c1a324a17b403e882a60546f64f7874eabb2e342" exitCode=0 Sep 11 14:07:26.711496 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:26.711264 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" event={"ID":"ff7145cb-d315-436a-867d-f62cb65214d0","Type":"ContainerDied","Data":"08a56eb0d1985ccd990490d6c1a324a17b403e882a60546f64f7874eabb2e342"} Sep 11 14:07:26.713819 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:26.713793 2566 generic.go:356] "Generic (PLEG): container finished" podID="7fc465ec-a28b-4185-ae4d-19e134e15d48" containerID="d08a1a4c4be19e18654e4e7e46c67d7d2aa97c471728eafa4046ebb80bc0059c" exitCode=0 Sep 11 14:07:26.713918 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:26.713829 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" event={"ID":"7fc465ec-a28b-4185-ae4d-19e134e15d48","Type":"ContainerDied","Data":"d08a1a4c4be19e18654e4e7e46c67d7d2aa97c471728eafa4046ebb80bc0059c"} Sep 11 14:07:27.721530 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:27.721498 2566 generic.go:356] "Generic (PLEG): container finished" podID="7fc465ec-a28b-4185-ae4d-19e134e15d48" containerID="a2b55004f933f36c266e669e5baf0b96ef0839a29a4611f756358533d3ff32d4" exitCode=0 Sep 11 14:07:27.721947 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:27.721579 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" event={"ID":"7fc465ec-a28b-4185-ae4d-19e134e15d48","Type":"ContainerDied","Data":"a2b55004f933f36c266e669e5baf0b96ef0839a29a4611f756358533d3ff32d4"} Sep 11 14:07:27.723366 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:27.723343 2566 generic.go:356] "Generic (PLEG): container finished" podID="4875ca2f-287a-4774-926f-84326befcabf" containerID="a9a9a6e461a63caa88724b2a8e6a79617184d150735576bebde1a5172ad66e00" exitCode=0 Sep 11 14:07:27.723468 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:27.723426 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" event={"ID":"4875ca2f-287a-4774-926f-84326befcabf","Type":"ContainerDied","Data":"a9a9a6e461a63caa88724b2a8e6a79617184d150735576bebde1a5172ad66e00"} Sep 11 14:07:27.725707 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:27.725688 2566 generic.go:356] "Generic (PLEG): container finished" podID="ff7145cb-d315-436a-867d-f62cb65214d0" containerID="76fa675b5da89564246e817a68074dc30b4b095bfac5c8b63c56ed3470072e12" exitCode=0 Sep 11 14:07:27.725795 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:27.725759 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" event={"ID":"ff7145cb-d315-436a-867d-f62cb65214d0","Type":"ContainerDied","Data":"76fa675b5da89564246e817a68074dc30b4b095bfac5c8b63c56ed3470072e12"} Sep 11 14:07:27.727610 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:27.727590 2566 generic.go:356] "Generic (PLEG): container finished" podID="0903f6d9-52a5-424c-954d-5296905d47d1" containerID="97fa0e036ebda0204d777c551a9d4a37daf207367a85a7e989d244c3bf1ca0c1" exitCode=0 Sep 11 14:07:27.727648 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:27.727634 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" event={"ID":"0903f6d9-52a5-424c-954d-5296905d47d1","Type":"ContainerDied","Data":"97fa0e036ebda0204d777c551a9d4a37daf207367a85a7e989d244c3bf1ca0c1"} Sep 11 14:07:28.734987 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:28.734955 2566 generic.go:356] "Generic (PLEG): container finished" podID="0903f6d9-52a5-424c-954d-5296905d47d1" containerID="ddb16de8cc2451ba1e3399be8b51181d831b9d4c719af320d251d12c33452c75" exitCode=0 Sep 11 14:07:28.735423 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:28.735055 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" event={"ID":"0903f6d9-52a5-424c-954d-5296905d47d1","Type":"ContainerDied","Data":"ddb16de8cc2451ba1e3399be8b51181d831b9d4c719af320d251d12c33452c75"} Sep 11 14:07:28.737209 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:28.737187 2566 generic.go:356] "Generic (PLEG): container finished" podID="4875ca2f-287a-4774-926f-84326befcabf" containerID="0bda0b6fd28468cd0f51d2926ca53c25e9dfaa154a54ecf4ff64f8256e407b73" exitCode=0 Sep 11 14:07:28.737307 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:28.737265 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" event={"ID":"4875ca2f-287a-4774-926f-84326befcabf","Type":"ContainerDied","Data":"0bda0b6fd28468cd0f51d2926ca53c25e9dfaa154a54ecf4ff64f8256e407b73"} Sep 11 14:07:28.867511 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:28.867482 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" Sep 11 14:07:28.902392 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:28.902361 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" Sep 11 14:07:28.954235 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:28.954201 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff7145cb-d315-436a-867d-f62cb65214d0-bundle\") pod \"ff7145cb-d315-436a-867d-f62cb65214d0\" (UID: \"ff7145cb-d315-436a-867d-f62cb65214d0\") " Sep 11 14:07:28.954428 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:28.954251 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xftsv\" (UniqueName: \"kubernetes.io/projected/ff7145cb-d315-436a-867d-f62cb65214d0-kube-api-access-xftsv\") pod \"ff7145cb-d315-436a-867d-f62cb65214d0\" (UID: \"ff7145cb-d315-436a-867d-f62cb65214d0\") " Sep 11 14:07:28.954428 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:28.954372 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff7145cb-d315-436a-867d-f62cb65214d0-util\") pod \"ff7145cb-d315-436a-867d-f62cb65214d0\" (UID: \"ff7145cb-d315-436a-867d-f62cb65214d0\") " Sep 11 14:07:28.954837 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:28.954812 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7145cb-d315-436a-867d-f62cb65214d0-bundle" (OuterVolumeSpecName: "bundle") pod "ff7145cb-d315-436a-867d-f62cb65214d0" (UID: "ff7145cb-d315-436a-867d-f62cb65214d0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:07:28.956619 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:28.956590 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7145cb-d315-436a-867d-f62cb65214d0-kube-api-access-xftsv" (OuterVolumeSpecName: "kube-api-access-xftsv") pod "ff7145cb-d315-436a-867d-f62cb65214d0" (UID: "ff7145cb-d315-436a-867d-f62cb65214d0"). InnerVolumeSpecName "kube-api-access-xftsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:07:28.959886 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:28.959867 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7145cb-d315-436a-867d-f62cb65214d0-util" (OuterVolumeSpecName: "util") pod "ff7145cb-d315-436a-867d-f62cb65214d0" (UID: "ff7145cb-d315-436a-867d-f62cb65214d0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:07:29.055351 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.055267 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fc465ec-a28b-4185-ae4d-19e134e15d48-bundle\") pod \"7fc465ec-a28b-4185-ae4d-19e134e15d48\" (UID: \"7fc465ec-a28b-4185-ae4d-19e134e15d48\") " Sep 11 14:07:29.055351 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.055301 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fc465ec-a28b-4185-ae4d-19e134e15d48-util\") pod \"7fc465ec-a28b-4185-ae4d-19e134e15d48\" (UID: \"7fc465ec-a28b-4185-ae4d-19e134e15d48\") " Sep 11 14:07:29.055351 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.055349 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgm25\" (UniqueName: \"kubernetes.io/projected/7fc465ec-a28b-4185-ae4d-19e134e15d48-kube-api-access-hgm25\") pod \"7fc465ec-a28b-4185-ae4d-19e134e15d48\" (UID: \"7fc465ec-a28b-4185-ae4d-19e134e15d48\") " Sep 11 14:07:29.055567 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.055485 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff7145cb-d315-436a-867d-f62cb65214d0-util\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:07:29.055567 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.055497 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff7145cb-d315-436a-867d-f62cb65214d0-bundle\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:07:29.055567 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.055509 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xftsv\" (UniqueName: \"kubernetes.io/projected/ff7145cb-d315-436a-867d-f62cb65214d0-kube-api-access-xftsv\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:07:29.055767 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.055743 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc465ec-a28b-4185-ae4d-19e134e15d48-bundle" (OuterVolumeSpecName: "bundle") pod "7fc465ec-a28b-4185-ae4d-19e134e15d48" (UID: "7fc465ec-a28b-4185-ae4d-19e134e15d48"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:07:29.057649 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.057618 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc465ec-a28b-4185-ae4d-19e134e15d48-kube-api-access-hgm25" (OuterVolumeSpecName: "kube-api-access-hgm25") pod "7fc465ec-a28b-4185-ae4d-19e134e15d48" (UID: "7fc465ec-a28b-4185-ae4d-19e134e15d48"). InnerVolumeSpecName "kube-api-access-hgm25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:07:29.060132 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.060111 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc465ec-a28b-4185-ae4d-19e134e15d48-util" (OuterVolumeSpecName: "util") pod "7fc465ec-a28b-4185-ae4d-19e134e15d48" (UID: "7fc465ec-a28b-4185-ae4d-19e134e15d48"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:07:29.156312 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.156274 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgm25\" (UniqueName: \"kubernetes.io/projected/7fc465ec-a28b-4185-ae4d-19e134e15d48-kube-api-access-hgm25\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:07:29.156312 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.156302 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fc465ec-a28b-4185-ae4d-19e134e15d48-bundle\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:07:29.156312 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.156311 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fc465ec-a28b-4185-ae4d-19e134e15d48-util\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:07:29.749295 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.749262 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" Sep 11 14:07:29.749295 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.749274 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7lkdgh" event={"ID":"7fc465ec-a28b-4185-ae4d-19e134e15d48","Type":"ContainerDied","Data":"a7927a969dabc810310411100b08a7fd9c0ae7170165786d025890eec52825bc"} Sep 11 14:07:29.749744 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.749319 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7927a969dabc810310411100b08a7fd9c0ae7170165786d025890eec52825bc" Sep 11 14:07:29.751372 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.751350 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" Sep 11 14:07:29.751372 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.751354 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7857r" event={"ID":"ff7145cb-d315-436a-867d-f62cb65214d0","Type":"ContainerDied","Data":"e3671dde3a4470e8c4c5313ede3b451ca8d8e4f0e1ec6ea360652fae8c7d14b4"} Sep 11 14:07:29.751546 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.751395 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3671dde3a4470e8c4c5313ede3b451ca8d8e4f0e1ec6ea360652fae8c7d14b4" Sep 11 14:07:29.907552 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.907530 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" Sep 11 14:07:29.909981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:29.909963 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" Sep 11 14:07:30.065314 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.065217 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4875ca2f-287a-4774-926f-84326befcabf-util\") pod \"4875ca2f-287a-4774-926f-84326befcabf\" (UID: \"4875ca2f-287a-4774-926f-84326befcabf\") " Sep 11 14:07:30.065314 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.065287 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kg25\" (UniqueName: \"kubernetes.io/projected/4875ca2f-287a-4774-926f-84326befcabf-kube-api-access-9kg25\") pod \"4875ca2f-287a-4774-926f-84326befcabf\" (UID: \"4875ca2f-287a-4774-926f-84326befcabf\") " Sep 11 14:07:30.065314 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.065318 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4875ca2f-287a-4774-926f-84326befcabf-bundle\") pod \"4875ca2f-287a-4774-926f-84326befcabf\" (UID: \"4875ca2f-287a-4774-926f-84326befcabf\") " Sep 11 14:07:30.065609 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.065345 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgccn\" (UniqueName: \"kubernetes.io/projected/0903f6d9-52a5-424c-954d-5296905d47d1-kube-api-access-pgccn\") pod \"0903f6d9-52a5-424c-954d-5296905d47d1\" (UID: \"0903f6d9-52a5-424c-954d-5296905d47d1\") " Sep 11 14:07:30.065609 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.065379 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0903f6d9-52a5-424c-954d-5296905d47d1-util\") pod \"0903f6d9-52a5-424c-954d-5296905d47d1\" (UID: \"0903f6d9-52a5-424c-954d-5296905d47d1\") " Sep 11 14:07:30.065609 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.065424 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0903f6d9-52a5-424c-954d-5296905d47d1-bundle\") pod \"0903f6d9-52a5-424c-954d-5296905d47d1\" (UID: \"0903f6d9-52a5-424c-954d-5296905d47d1\") " Sep 11 14:07:30.065881 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.065848 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4875ca2f-287a-4774-926f-84326befcabf-bundle" (OuterVolumeSpecName: "bundle") pod "4875ca2f-287a-4774-926f-84326befcabf" (UID: "4875ca2f-287a-4774-926f-84326befcabf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:07:30.066105 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.066022 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0903f6d9-52a5-424c-954d-5296905d47d1-bundle" (OuterVolumeSpecName: "bundle") pod "0903f6d9-52a5-424c-954d-5296905d47d1" (UID: "0903f6d9-52a5-424c-954d-5296905d47d1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:07:30.067665 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.067642 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0903f6d9-52a5-424c-954d-5296905d47d1-kube-api-access-pgccn" (OuterVolumeSpecName: "kube-api-access-pgccn") pod "0903f6d9-52a5-424c-954d-5296905d47d1" (UID: "0903f6d9-52a5-424c-954d-5296905d47d1"). InnerVolumeSpecName "kube-api-access-pgccn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:07:30.067776 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.067761 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4875ca2f-287a-4774-926f-84326befcabf-kube-api-access-9kg25" (OuterVolumeSpecName: "kube-api-access-9kg25") pod "4875ca2f-287a-4774-926f-84326befcabf" (UID: "4875ca2f-287a-4774-926f-84326befcabf"). InnerVolumeSpecName "kube-api-access-9kg25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:07:30.071346 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.071326 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0903f6d9-52a5-424c-954d-5296905d47d1-util" (OuterVolumeSpecName: "util") pod "0903f6d9-52a5-424c-954d-5296905d47d1" (UID: "0903f6d9-52a5-424c-954d-5296905d47d1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:07:30.073316 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.073294 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4875ca2f-287a-4774-926f-84326befcabf-util" (OuterVolumeSpecName: "util") pod "4875ca2f-287a-4774-926f-84326befcabf" (UID: "4875ca2f-287a-4774-926f-84326befcabf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:07:30.166803 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.166765 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgccn\" (UniqueName: \"kubernetes.io/projected/0903f6d9-52a5-424c-954d-5296905d47d1-kube-api-access-pgccn\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:07:30.166803 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.166802 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0903f6d9-52a5-424c-954d-5296905d47d1-util\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:07:30.166803 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.166812 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0903f6d9-52a5-424c-954d-5296905d47d1-bundle\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:07:30.166803 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.166819 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4875ca2f-287a-4774-926f-84326befcabf-util\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:07:30.167061 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.166828 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kg25\" (UniqueName: \"kubernetes.io/projected/4875ca2f-287a-4774-926f-84326befcabf-kube-api-access-9kg25\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:07:30.167061 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.166836 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4875ca2f-287a-4774-926f-84326befcabf-bundle\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:07:30.758962 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.758937 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" Sep 11 14:07:30.758962 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.758935 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6t248c" event={"ID":"4875ca2f-287a-4774-926f-84326befcabf","Type":"ContainerDied","Data":"2cab24ed4b6745ef2b71f8cc332a614cd4f8942f36205eb913bdc2ce0174320c"} Sep 11 14:07:30.759518 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.758998 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cab24ed4b6745ef2b71f8cc332a614cd4f8942f36205eb913bdc2ce0174320c" Sep 11 14:07:30.761063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.761039 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" event={"ID":"0903f6d9-52a5-424c-954d-5296905d47d1","Type":"ContainerDied","Data":"046609d3eef743c3333544bc98b88e6c431bbde59df59049dbf0d0865e55c688"} Sep 11 14:07:30.761219 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.761115 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2c5sn" Sep 11 14:07:30.761219 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:30.761138 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="046609d3eef743c3333544bc98b88e6c431bbde59df59049dbf0d0865e55c688" Sep 11 14:07:38.971686 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.971654 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5"] Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.971975 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0903f6d9-52a5-424c-954d-5296905d47d1" containerName="extract" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.971986 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0903f6d9-52a5-424c-954d-5296905d47d1" containerName="extract" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.971997 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff7145cb-d315-436a-867d-f62cb65214d0" containerName="pull" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972003 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff7145cb-d315-436a-867d-f62cb65214d0" containerName="pull" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972011 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0903f6d9-52a5-424c-954d-5296905d47d1" containerName="pull" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972016 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0903f6d9-52a5-424c-954d-5296905d47d1" containerName="pull" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972026 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff7145cb-d315-436a-867d-f62cb65214d0" containerName="util" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972032 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff7145cb-d315-436a-867d-f62cb65214d0" containerName="util" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972038 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff7145cb-d315-436a-867d-f62cb65214d0" containerName="extract" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972043 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff7145cb-d315-436a-867d-f62cb65214d0" containerName="extract" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972049 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fc465ec-a28b-4185-ae4d-19e134e15d48" containerName="pull" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972054 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fc465ec-a28b-4185-ae4d-19e134e15d48" containerName="pull" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972059 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fc465ec-a28b-4185-ae4d-19e134e15d48" containerName="extract" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972064 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fc465ec-a28b-4185-ae4d-19e134e15d48" containerName="extract" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972070 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0903f6d9-52a5-424c-954d-5296905d47d1" containerName="util" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972074 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0903f6d9-52a5-424c-954d-5296905d47d1" containerName="util" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972079 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4875ca2f-287a-4774-926f-84326befcabf" containerName="extract" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972098 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4875ca2f-287a-4774-926f-84326befcabf" containerName="extract" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972105 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4875ca2f-287a-4774-926f-84326befcabf" containerName="util" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972110 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4875ca2f-287a-4774-926f-84326befcabf" containerName="util" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972117 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fc465ec-a28b-4185-ae4d-19e134e15d48" containerName="util" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972121 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fc465ec-a28b-4185-ae4d-19e134e15d48" containerName="util" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972128 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4875ca2f-287a-4774-926f-84326befcabf" containerName="pull" Sep 11 14:07:38.972164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972134 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4875ca2f-287a-4774-926f-84326befcabf" containerName="pull" Sep 11 14:07:38.972851 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972198 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff7145cb-d315-436a-867d-f62cb65214d0" containerName="extract" Sep 11 14:07:38.972851 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972206 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0903f6d9-52a5-424c-954d-5296905d47d1" containerName="extract" Sep 11 14:07:38.972851 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972214 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fc465ec-a28b-4185-ae4d-19e134e15d48" containerName="extract" Sep 11 14:07:38.972851 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.972220 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4875ca2f-287a-4774-926f-84326befcabf" containerName="extract" Sep 11 14:07:38.975284 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.975270 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:38.977753 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.977727 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l9xxt\"" Sep 11 14:07:38.978313 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.978290 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 14:07:38.978418 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.978345 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 14:07:38.993847 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:38.993820 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5"] Sep 11 14:07:39.147213 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:39.147179 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e99be3e4-933d-4712-b097-ff7787a4e603-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" (UID: \"e99be3e4-933d-4712-b097-ff7787a4e603\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:39.147387 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:39.147247 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tdcl\" (UniqueName: \"kubernetes.io/projected/e99be3e4-933d-4712-b097-ff7787a4e603-kube-api-access-8tdcl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" (UID: \"e99be3e4-933d-4712-b097-ff7787a4e603\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:39.248537 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:39.248502 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e99be3e4-933d-4712-b097-ff7787a4e603-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" (UID: \"e99be3e4-933d-4712-b097-ff7787a4e603\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:39.248707 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:39.248564 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tdcl\" (UniqueName: \"kubernetes.io/projected/e99be3e4-933d-4712-b097-ff7787a4e603-kube-api-access-8tdcl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" (UID: \"e99be3e4-933d-4712-b097-ff7787a4e603\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:39.248950 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:39.248927 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e99be3e4-933d-4712-b097-ff7787a4e603-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" (UID: \"e99be3e4-933d-4712-b097-ff7787a4e603\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:39.266942 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:39.266918 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tdcl\" (UniqueName: \"kubernetes.io/projected/e99be3e4-933d-4712-b097-ff7787a4e603-kube-api-access-8tdcl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" (UID: \"e99be3e4-933d-4712-b097-ff7787a4e603\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:39.285762 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:39.285735 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:39.576626 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:39.576566 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5"] Sep 11 14:07:39.578967 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:07:39.578940 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99be3e4_933d_4712_b097_ff7787a4e603.slice/crio-5d02e347c00f29b48529bdd1868ce900b69e0610d281fd2bfc16e70fba7a980f WatchSource:0}: Error finding container 5d02e347c00f29b48529bdd1868ce900b69e0610d281fd2bfc16e70fba7a980f: Status 404 returned error can't find the container with id 5d02e347c00f29b48529bdd1868ce900b69e0610d281fd2bfc16e70fba7a980f Sep 11 14:07:39.811582 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:39.811546 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" event={"ID":"e99be3e4-933d-4712-b097-ff7787a4e603","Type":"ContainerStarted","Data":"5d02e347c00f29b48529bdd1868ce900b69e0610d281fd2bfc16e70fba7a980f"} Sep 11 14:07:40.060716 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:40.060689 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:07:40.068128 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:40.068079 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:07:41.861896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:41.861860 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-cw6lr"] Sep 11 14:07:41.871423 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:41.871385 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-cw6lr" Sep 11 14:07:41.874595 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:41.874561 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-j4xmt\"" Sep 11 14:07:41.880954 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:41.880924 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-cw6lr"] Sep 11 14:07:41.972844 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:41.972796 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7tbn\" (UniqueName: \"kubernetes.io/projected/44ea994f-1c5d-4fa0-8029-14c347f2df55-kube-api-access-z7tbn\") pod \"authorino-operator-76b478bd78-cw6lr\" (UID: \"44ea994f-1c5d-4fa0-8029-14c347f2df55\") " pod="kuadrant-system/authorino-operator-76b478bd78-cw6lr" Sep 11 14:07:42.073834 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:42.073783 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7tbn\" (UniqueName: \"kubernetes.io/projected/44ea994f-1c5d-4fa0-8029-14c347f2df55-kube-api-access-z7tbn\") pod \"authorino-operator-76b478bd78-cw6lr\" (UID: \"44ea994f-1c5d-4fa0-8029-14c347f2df55\") " pod="kuadrant-system/authorino-operator-76b478bd78-cw6lr" Sep 11 14:07:42.118880 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:42.118775 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7tbn\" (UniqueName: \"kubernetes.io/projected/44ea994f-1c5d-4fa0-8029-14c347f2df55-kube-api-access-z7tbn\") pod \"authorino-operator-76b478bd78-cw6lr\" (UID: \"44ea994f-1c5d-4fa0-8029-14c347f2df55\") " pod="kuadrant-system/authorino-operator-76b478bd78-cw6lr" Sep 11 14:07:42.186924 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:42.186889 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-cw6lr" Sep 11 14:07:42.515780 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:42.515750 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-cw6lr"] Sep 11 14:07:42.516784 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:07:42.516755 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ea994f_1c5d_4fa0_8029_14c347f2df55.slice/crio-d79481ccf1a19a9785ef30fbc0e891f2b3560cee0cfe70da115c78ed2f052246 WatchSource:0}: Error finding container d79481ccf1a19a9785ef30fbc0e891f2b3560cee0cfe70da115c78ed2f052246: Status 404 returned error can't find the container with id d79481ccf1a19a9785ef30fbc0e891f2b3560cee0cfe70da115c78ed2f052246 Sep 11 14:07:42.833737 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:42.833656 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-cw6lr" event={"ID":"44ea994f-1c5d-4fa0-8029-14c347f2df55","Type":"ContainerStarted","Data":"d79481ccf1a19a9785ef30fbc0e891f2b3560cee0cfe70da115c78ed2f052246"} Sep 11 14:07:43.827477 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:43.827403 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j"] Sep 11 14:07:43.830954 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:43.830927 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:43.833929 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:43.833902 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xfzw2\"" Sep 11 14:07:43.848443 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:43.848421 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j"] Sep 11 14:07:43.891435 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:43.891405 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8kwc\" (UniqueName: \"kubernetes.io/projected/bb87dc4e-6607-4be5-b6c4-00524ffd44d0-kube-api-access-w8kwc\") pod \"limitador-operator-controller-manager-74495c69f8-9mt2j\" (UID: \"bb87dc4e-6607-4be5-b6c4-00524ffd44d0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:43.992456 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:43.992421 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8kwc\" (UniqueName: \"kubernetes.io/projected/bb87dc4e-6607-4be5-b6c4-00524ffd44d0-kube-api-access-w8kwc\") pod \"limitador-operator-controller-manager-74495c69f8-9mt2j\" (UID: \"bb87dc4e-6607-4be5-b6c4-00524ffd44d0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:44.002493 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:44.002468 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8kwc\" (UniqueName: \"kubernetes.io/projected/bb87dc4e-6607-4be5-b6c4-00524ffd44d0-kube-api-access-w8kwc\") pod \"limitador-operator-controller-manager-74495c69f8-9mt2j\" (UID: \"bb87dc4e-6607-4be5-b6c4-00524ffd44d0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:44.142202 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:44.142109 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:44.428721 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:44.428689 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j"] Sep 11 14:07:44.491298 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:07:44.491259 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb87dc4e_6607_4be5_b6c4_00524ffd44d0.slice/crio-a9f2e52d368c7ecd91399df2a711d4901955fd91a46a71aaac92ba2f9d442812 WatchSource:0}: Error finding container a9f2e52d368c7ecd91399df2a711d4901955fd91a46a71aaac92ba2f9d442812: Status 404 returned error can't find the container with id a9f2e52d368c7ecd91399df2a711d4901955fd91a46a71aaac92ba2f9d442812 Sep 11 14:07:44.846739 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:44.846698 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" event={"ID":"bb87dc4e-6607-4be5-b6c4-00524ffd44d0","Type":"ContainerStarted","Data":"a9f2e52d368c7ecd91399df2a711d4901955fd91a46a71aaac92ba2f9d442812"} Sep 11 14:07:44.848515 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:44.848489 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-cw6lr" event={"ID":"44ea994f-1c5d-4fa0-8029-14c347f2df55","Type":"ContainerStarted","Data":"a1016bc9c3c4ef38f5604419d402a8b2bada6a61b09d81beb73f595d3a5ffbc2"} Sep 11 14:07:44.848680 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:44.848652 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-cw6lr" Sep 11 14:07:44.850410 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:44.850383 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" event={"ID":"e99be3e4-933d-4712-b097-ff7787a4e603","Type":"ContainerStarted","Data":"d8c37ecfecbdc8d7227341eb16a7aae4f542d6ef295c9de39a0bd9b19f721934"} Sep 11 14:07:44.850555 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:44.850446 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:44.904055 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:44.904005 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-cw6lr" podStartSLOduration=1.870512478 podStartE2EDuration="3.903989907s" totalImagesPullingTime="2.033477429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:44.897661685 +0000 UTC m=+605.302642823" watchObservedRunningTime="2026-09-11 14:07:44.903989907 +0000 UTC m=+605.308971032" Sep 11 14:07:44.949092 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:44.949047 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" podStartSLOduration=2.681451869 podStartE2EDuration="6.949031255s" totalImagesPullingTime="4.267579386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:44.945701292 +0000 UTC m=+605.350682431" watchObservedRunningTime="2026-09-11 14:07:44.949031255 +0000 UTC m=+605.354012383" Sep 11 14:07:46.866884 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:46.866851 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" event={"ID":"bb87dc4e-6607-4be5-b6c4-00524ffd44d0","Type":"ContainerStarted","Data":"a2daab04b430e177ea2b6330261975b33ca7b77612b94dbd0ddde3b92ebb4462"} Sep 11 14:07:46.867336 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:46.866909 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:46.897191 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:46.897143 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" podStartSLOduration=2.133168479 podStartE2EDuration="3.897127302s" totalImagesPullingTime="1.763958823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:46.895047619 +0000 UTC m=+607.300028759" watchObservedRunningTime="2026-09-11 14:07:46.897127302 +0000 UTC m=+607.302108430" Sep 11 14:07:55.862207 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:55.862175 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:55.864960 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:55.862259 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-cw6lr" Sep 11 14:07:57.875399 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:57.875362 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:58.226430 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.226389 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5"] Sep 11 14:07:58.226650 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.226623 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" containerName="manager" containerID="cri-o://d8c37ecfecbdc8d7227341eb16a7aae4f542d6ef295c9de39a0bd9b19f721934" gracePeriod=2 Sep 11 14:07:58.240978 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.240948 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5"] Sep 11 14:07:58.248781 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.248738 2566 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"e99be3e4-933d-4712-b097-ff7787a4e603\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\": pods \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:58.256479 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.256456 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j"] Sep 11 14:07:58.256665 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.256643 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" containerName="manager" containerID="cri-o://a2daab04b430e177ea2b6330261975b33ca7b77612b94dbd0ddde3b92ebb4462" gracePeriod=2 Sep 11 14:07:58.267925 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.267898 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw"] Sep 11 14:07:58.268457 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.268434 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" containerName="manager" Sep 11 14:07:58.268611 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.268598 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" containerName="manager" Sep 11 14:07:58.268820 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.268791 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" containerName="manager" Sep 11 14:07:58.269260 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.268435 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:58.271800 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.271781 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j"] Sep 11 14:07:58.271902 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.271882 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" Sep 11 14:07:58.274232 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.274200 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9mt2j\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:58.294220 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.294190 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw"] Sep 11 14:07:58.296711 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.296682 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vpfjn"] Sep 11 14:07:58.297065 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.297048 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" containerName="manager" Sep 11 14:07:58.297065 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.297063 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" containerName="manager" Sep 11 14:07:58.297248 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.297151 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" containerName="manager" Sep 11 14:07:58.300100 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.300065 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vpfjn" Sep 11 14:07:58.318774 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.318749 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vpfjn"] Sep 11 14:07:58.323049 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.323023 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79bc82ee-1165-4e1c-a930-d41745214818-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wxwhw\" (UID: \"79bc82ee-1165-4e1c-a930-d41745214818\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" Sep 11 14:07:58.323155 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.323135 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6qp6\" (UniqueName: \"kubernetes.io/projected/79bc82ee-1165-4e1c-a930-d41745214818-kube-api-access-q6qp6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wxwhw\" (UID: \"79bc82ee-1165-4e1c-a930-d41745214818\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" Sep 11 14:07:58.332364 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.332338 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:58.334416 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.334386 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:58.337623 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.337601 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9mt2j\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:58.424722 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.424676 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qp6\" (UniqueName: \"kubernetes.io/projected/79bc82ee-1165-4e1c-a930-d41745214818-kube-api-access-q6qp6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wxwhw\" (UID: \"79bc82ee-1165-4e1c-a930-d41745214818\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" Sep 11 14:07:58.424859 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.424776 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx8qv\" (UniqueName: \"kubernetes.io/projected/ffcecabb-7b78-4fb8-99ed-3918f9b1db5b-kube-api-access-wx8qv\") pod \"limitador-operator-controller-manager-74495c69f8-vpfjn\" (UID: \"ffcecabb-7b78-4fb8-99ed-3918f9b1db5b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vpfjn" Sep 11 14:07:58.424859 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.424820 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79bc82ee-1165-4e1c-a930-d41745214818-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wxwhw\" (UID: \"79bc82ee-1165-4e1c-a930-d41745214818\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" Sep 11 14:07:58.425280 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.425256 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79bc82ee-1165-4e1c-a930-d41745214818-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wxwhw\" (UID: \"79bc82ee-1165-4e1c-a930-d41745214818\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" Sep 11 14:07:58.442500 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.442465 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6qp6\" (UniqueName: \"kubernetes.io/projected/79bc82ee-1165-4e1c-a930-d41745214818-kube-api-access-q6qp6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wxwhw\" (UID: \"79bc82ee-1165-4e1c-a930-d41745214818\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" Sep 11 14:07:58.498776 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.498756 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:58.500798 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.500777 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:58.500999 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.500978 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9mt2j\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:58.503272 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.503249 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:58.505472 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.505451 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9mt2j\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:58.507525 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.507505 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:58.525914 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.525889 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx8qv\" (UniqueName: \"kubernetes.io/projected/ffcecabb-7b78-4fb8-99ed-3918f9b1db5b-kube-api-access-wx8qv\") pod \"limitador-operator-controller-manager-74495c69f8-vpfjn\" (UID: \"ffcecabb-7b78-4fb8-99ed-3918f9b1db5b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vpfjn" Sep 11 14:07:58.540062 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.540033 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx8qv\" (UniqueName: \"kubernetes.io/projected/ffcecabb-7b78-4fb8-99ed-3918f9b1db5b-kube-api-access-wx8qv\") pod \"limitador-operator-controller-manager-74495c69f8-vpfjn\" (UID: \"ffcecabb-7b78-4fb8-99ed-3918f9b1db5b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vpfjn" Sep 11 14:07:58.627187 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.627155 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8kwc\" (UniqueName: \"kubernetes.io/projected/bb87dc4e-6607-4be5-b6c4-00524ffd44d0-kube-api-access-w8kwc\") pod \"bb87dc4e-6607-4be5-b6c4-00524ffd44d0\" (UID: \"bb87dc4e-6607-4be5-b6c4-00524ffd44d0\") " Sep 11 14:07:58.627348 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.627200 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tdcl\" (UniqueName: \"kubernetes.io/projected/e99be3e4-933d-4712-b097-ff7787a4e603-kube-api-access-8tdcl\") pod \"e99be3e4-933d-4712-b097-ff7787a4e603\" (UID: \"e99be3e4-933d-4712-b097-ff7787a4e603\") " Sep 11 14:07:58.627348 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.627245 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e99be3e4-933d-4712-b097-ff7787a4e603-extensions-socket-volume\") pod \"e99be3e4-933d-4712-b097-ff7787a4e603\" (UID: \"e99be3e4-933d-4712-b097-ff7787a4e603\") " Sep 11 14:07:58.627800 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.627769 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99be3e4-933d-4712-b097-ff7787a4e603-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e99be3e4-933d-4712-b097-ff7787a4e603" (UID: "e99be3e4-933d-4712-b097-ff7787a4e603"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:07:58.629465 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.629440 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99be3e4-933d-4712-b097-ff7787a4e603-kube-api-access-8tdcl" (OuterVolumeSpecName: "kube-api-access-8tdcl") pod "e99be3e4-933d-4712-b097-ff7787a4e603" (UID: "e99be3e4-933d-4712-b097-ff7787a4e603"). InnerVolumeSpecName "kube-api-access-8tdcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:07:58.629547 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.629465 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb87dc4e-6607-4be5-b6c4-00524ffd44d0-kube-api-access-w8kwc" (OuterVolumeSpecName: "kube-api-access-w8kwc") pod "bb87dc4e-6607-4be5-b6c4-00524ffd44d0" (UID: "bb87dc4e-6607-4be5-b6c4-00524ffd44d0"). InnerVolumeSpecName "kube-api-access-w8kwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:07:58.675683 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.675644 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" Sep 11 14:07:58.681795 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.681772 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vpfjn" Sep 11 14:07:58.728599 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.728466 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8kwc\" (UniqueName: \"kubernetes.io/projected/bb87dc4e-6607-4be5-b6c4-00524ffd44d0-kube-api-access-w8kwc\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:07:58.728599 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.728505 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tdcl\" (UniqueName: \"kubernetes.io/projected/e99be3e4-933d-4712-b097-ff7787a4e603-kube-api-access-8tdcl\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:07:58.728599 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.728534 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e99be3e4-933d-4712-b097-ff7787a4e603-extensions-socket-volume\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:07:58.932132 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.932032 2566 generic.go:356] "Generic (PLEG): container finished" podID="e99be3e4-933d-4712-b097-ff7787a4e603" containerID="d8c37ecfecbdc8d7227341eb16a7aae4f542d6ef295c9de39a0bd9b19f721934" exitCode=0 Sep 11 14:07:58.932577 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.932139 2566 scope.go:117] "RemoveContainer" containerID="d8c37ecfecbdc8d7227341eb16a7aae4f542d6ef295c9de39a0bd9b19f721934" Sep 11 14:07:58.932577 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.932109 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:58.934069 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.933868 2566 generic.go:356] "Generic (PLEG): container finished" podID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" containerID="a2daab04b430e177ea2b6330261975b33ca7b77612b94dbd0ddde3b92ebb4462" exitCode=0 Sep 11 14:07:58.934069 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.933947 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:58.935417 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.935391 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9mt2j\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:58.937819 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.937790 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:58.940261 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.940228 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9mt2j\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:58.942443 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.942414 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:58.943044 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.943025 2566 scope.go:117] "RemoveContainer" containerID="d8c37ecfecbdc8d7227341eb16a7aae4f542d6ef295c9de39a0bd9b19f721934" Sep 11 14:07:58.943508 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.943409 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c37ecfecbdc8d7227341eb16a7aae4f542d6ef295c9de39a0bd9b19f721934\": container with ID starting with d8c37ecfecbdc8d7227341eb16a7aae4f542d6ef295c9de39a0bd9b19f721934 not found: ID does not exist" containerID="d8c37ecfecbdc8d7227341eb16a7aae4f542d6ef295c9de39a0bd9b19f721934" Sep 11 14:07:58.943508 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.943435 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c37ecfecbdc8d7227341eb16a7aae4f542d6ef295c9de39a0bd9b19f721934"} err="failed to get container status \"d8c37ecfecbdc8d7227341eb16a7aae4f542d6ef295c9de39a0bd9b19f721934\": rpc error: code = NotFound desc = could not find container \"d8c37ecfecbdc8d7227341eb16a7aae4f542d6ef295c9de39a0bd9b19f721934\": container with ID starting with d8c37ecfecbdc8d7227341eb16a7aae4f542d6ef295c9de39a0bd9b19f721934 not found: ID does not exist" Sep 11 14:07:58.943508 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.943451 2566 scope.go:117] "RemoveContainer" containerID="a2daab04b430e177ea2b6330261975b33ca7b77612b94dbd0ddde3b92ebb4462" Sep 11 14:07:58.947847 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.947819 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9mt2j\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:58.949936 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.949910 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:58.952053 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.952037 2566 scope.go:117] "RemoveContainer" containerID="a2daab04b430e177ea2b6330261975b33ca7b77612b94dbd0ddde3b92ebb4462" Sep 11 14:07:58.952345 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.952327 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2daab04b430e177ea2b6330261975b33ca7b77612b94dbd0ddde3b92ebb4462\": container with ID starting with a2daab04b430e177ea2b6330261975b33ca7b77612b94dbd0ddde3b92ebb4462 not found: ID does not exist" containerID="a2daab04b430e177ea2b6330261975b33ca7b77612b94dbd0ddde3b92ebb4462" Sep 11 14:07:58.952446 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.952352 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2daab04b430e177ea2b6330261975b33ca7b77612b94dbd0ddde3b92ebb4462"} err="failed to get container status \"a2daab04b430e177ea2b6330261975b33ca7b77612b94dbd0ddde3b92ebb4462\": rpc error: code = NotFound desc = could not find container \"a2daab04b430e177ea2b6330261975b33ca7b77612b94dbd0ddde3b92ebb4462\": container with ID starting with a2daab04b430e177ea2b6330261975b33ca7b77612b94dbd0ddde3b92ebb4462 not found: ID does not exist" Sep 11 14:07:58.952446 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.952438 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9mt2j\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:58.954949 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:58.954932 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:58.985221 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:07:58.985183 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bc82ee_1165_4e1c_a930_d41745214818.slice/crio-958d8c76c7b380c69faa9ec44c5383e065c8a2e2a7bf1b5d6c3ccd2d083ac42a WatchSource:0}: Error finding container 958d8c76c7b380c69faa9ec44c5383e065c8a2e2a7bf1b5d6c3ccd2d083ac42a: Status 404 returned error can't find the container with id 958d8c76c7b380c69faa9ec44c5383e065c8a2e2a7bf1b5d6c3ccd2d083ac42a Sep 11 14:07:58.987522 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.987456 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw"] Sep 11 14:07:58.991746 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:58.991727 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vpfjn"] Sep 11 14:07:58.993949 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:07:58.993930 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcecabb_7b78_4fb8_99ed_3918f9b1db5b.slice/crio-6bab5f147ba6159e9c64a4f1a9b79c8f2c23f070d43450f411f72b42df83a44d WatchSource:0}: Error finding container 6bab5f147ba6159e9c64a4f1a9b79c8f2c23f070d43450f411f72b42df83a44d: Status 404 returned error can't find the container with id 6bab5f147ba6159e9c64a4f1a9b79c8f2c23f070d43450f411f72b42df83a44d Sep 11 14:07:59.750331 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.750300 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd"] Sep 11 14:07:59.753815 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.753798 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" Sep 11 14:07:59.756192 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:59.756163 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9mt2j\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:59.758338 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:59.758303 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:59.787948 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.787923 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd"] Sep 11 14:07:59.838112 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.838056 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b6qk\" (UniqueName: \"kubernetes.io/projected/264ffa70-1a57-4574-8e9c-7803d5260a37-kube-api-access-8b6qk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v9csd\" (UID: \"264ffa70-1a57-4574-8e9c-7803d5260a37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" Sep 11 14:07:59.838270 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.838124 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/264ffa70-1a57-4574-8e9c-7803d5260a37-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v9csd\" (UID: \"264ffa70-1a57-4574-8e9c-7803d5260a37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" Sep 11 14:07:59.938627 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.938601 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b6qk\" (UniqueName: \"kubernetes.io/projected/264ffa70-1a57-4574-8e9c-7803d5260a37-kube-api-access-8b6qk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v9csd\" (UID: \"264ffa70-1a57-4574-8e9c-7803d5260a37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" Sep 11 14:07:59.938982 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.938637 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/264ffa70-1a57-4574-8e9c-7803d5260a37-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v9csd\" (UID: \"264ffa70-1a57-4574-8e9c-7803d5260a37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" Sep 11 14:07:59.939042 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.939025 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/264ffa70-1a57-4574-8e9c-7803d5260a37-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v9csd\" (UID: \"264ffa70-1a57-4574-8e9c-7803d5260a37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" Sep 11 14:07:59.941535 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.941508 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vpfjn" event={"ID":"ffcecabb-7b78-4fb8-99ed-3918f9b1db5b","Type":"ContainerStarted","Data":"51f5fe5a39710447471ceae83c50c8f49705b102b949de9cdb205cb2408106c1"} Sep 11 14:07:59.941681 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.941555 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vpfjn" event={"ID":"ffcecabb-7b78-4fb8-99ed-3918f9b1db5b","Type":"ContainerStarted","Data":"6bab5f147ba6159e9c64a4f1a9b79c8f2c23f070d43450f411f72b42df83a44d"} Sep 11 14:07:59.941681 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.941601 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vpfjn" Sep 11 14:07:59.943673 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:59.943646 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9mt2j\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:07:59.944149 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.944123 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" event={"ID":"79bc82ee-1165-4e1c-a930-d41745214818","Type":"ContainerStarted","Data":"47fd3f13b8c0b262e5db84afb7bc35445928c2e3bf321b261c282e40be701482"} Sep 11 14:07:59.944253 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.944166 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" event={"ID":"79bc82ee-1165-4e1c-a930-d41745214818","Type":"ContainerStarted","Data":"958d8c76c7b380c69faa9ec44c5383e065c8a2e2a7bf1b5d6c3ccd2d083ac42a"} Sep 11 14:07:59.944253 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.944235 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" Sep 11 14:07:59.953163 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.953141 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b6qk\" (UniqueName: \"kubernetes.io/projected/264ffa70-1a57-4574-8e9c-7803d5260a37-kube-api-access-8b6qk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v9csd\" (UID: \"264ffa70-1a57-4574-8e9c-7803d5260a37\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" Sep 11 14:07:59.979666 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:07:59.979625 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vpfjn" podStartSLOduration=1.979610243 podStartE2EDuration="1.979610243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:59.978260113 +0000 UTC m=+620.383241240" watchObservedRunningTime="2026-09-11 14:07:59.979610243 +0000 UTC m=+620.384591371" Sep 11 14:07:59.980794 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:59.980774 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:07:59.982900 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:07:59.982876 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9mt2j\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:08:00.011728 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:08:00.011642 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:08:00.012148 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:00.012109 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" podStartSLOduration=2.012072851 podStartE2EDuration="2.012072851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:00.009495458 +0000 UTC m=+620.414476601" watchObservedRunningTime="2026-09-11 14:08:00.012072851 +0000 UTC m=+620.417053978" Sep 11 14:08:00.064032 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:00.063993 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" Sep 11 14:08:00.096943 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:08:00.096904 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-gbhr5\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gbhr5" Sep 11 14:08:00.098579 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:00.098538 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" path="/var/lib/kubelet/pods/bb87dc4e-6607-4be5-b6c4-00524ffd44d0/volumes" Sep 11 14:08:00.099101 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:00.099062 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e99be3e4-933d-4712-b097-ff7787a4e603" path="/var/lib/kubelet/pods/e99be3e4-933d-4712-b097-ff7787a4e603/volumes" Sep 11 14:08:00.099423 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:08:00.099385 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9mt2j\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="bb87dc4e-6607-4be5-b6c4-00524ffd44d0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9mt2j" Sep 11 14:08:00.407319 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:00.407281 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd"] Sep 11 14:08:00.408148 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:08:00.408117 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264ffa70_1a57_4574_8e9c_7803d5260a37.slice/crio-735dbd783f96e9baab2a65da449f4289998b095b771e377776d3901acb9fa41e WatchSource:0}: Error finding container 735dbd783f96e9baab2a65da449f4289998b095b771e377776d3901acb9fa41e: Status 404 returned error can't find the container with id 735dbd783f96e9baab2a65da449f4289998b095b771e377776d3901acb9fa41e Sep 11 14:08:00.953508 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:00.953474 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" event={"ID":"264ffa70-1a57-4574-8e9c-7803d5260a37","Type":"ContainerStarted","Data":"dc9959f0d0f00d058dcf2aa2bedcd27e1ce1c131440cb0c9ad4c5ba4696c0d90"} Sep 11 14:08:00.953928 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:00.953539 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" event={"ID":"264ffa70-1a57-4574-8e9c-7803d5260a37","Type":"ContainerStarted","Data":"735dbd783f96e9baab2a65da449f4289998b095b771e377776d3901acb9fa41e"} Sep 11 14:08:00.986661 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:00.986615 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" podStartSLOduration=1.98659867 podStartE2EDuration="1.98659867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:00.984000425 +0000 UTC m=+621.388981552" watchObservedRunningTime="2026-09-11 14:08:00.98659867 +0000 UTC m=+621.391579798" Sep 11 14:08:01.958633 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:01.958559 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" Sep 11 14:08:10.955297 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:10.955263 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vpfjn" Sep 11 14:08:10.955795 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:10.955498 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" Sep 11 14:08:12.967866 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:12.967833 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" Sep 11 14:08:13.034425 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:13.034393 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw"] Sep 11 14:08:13.034641 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:13.034618 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" podUID="79bc82ee-1165-4e1c-a930-d41745214818" containerName="manager" containerID="cri-o://47fd3f13b8c0b262e5db84afb7bc35445928c2e3bf321b261c282e40be701482" gracePeriod=10 Sep 11 14:08:13.291403 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:13.291380 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" Sep 11 14:08:13.350818 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:13.350790 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79bc82ee-1165-4e1c-a930-d41745214818-extensions-socket-volume\") pod \"79bc82ee-1165-4e1c-a930-d41745214818\" (UID: \"79bc82ee-1165-4e1c-a930-d41745214818\") " Sep 11 14:08:13.351018 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:13.350849 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6qp6\" (UniqueName: \"kubernetes.io/projected/79bc82ee-1165-4e1c-a930-d41745214818-kube-api-access-q6qp6\") pod \"79bc82ee-1165-4e1c-a930-d41745214818\" (UID: \"79bc82ee-1165-4e1c-a930-d41745214818\") " Sep 11 14:08:13.351178 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:13.351154 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bc82ee-1165-4e1c-a930-d41745214818-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "79bc82ee-1165-4e1c-a930-d41745214818" (UID: "79bc82ee-1165-4e1c-a930-d41745214818"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:08:13.353228 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:13.353201 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bc82ee-1165-4e1c-a930-d41745214818-kube-api-access-q6qp6" (OuterVolumeSpecName: "kube-api-access-q6qp6") pod "79bc82ee-1165-4e1c-a930-d41745214818" (UID: "79bc82ee-1165-4e1c-a930-d41745214818"). InnerVolumeSpecName "kube-api-access-q6qp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:08:13.451646 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:13.451619 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79bc82ee-1165-4e1c-a930-d41745214818-extensions-socket-volume\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:08:13.451646 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:13.451642 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6qp6\" (UniqueName: \"kubernetes.io/projected/79bc82ee-1165-4e1c-a930-d41745214818-kube-api-access-q6qp6\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:08:14.031913 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:14.031878 2566 generic.go:356] "Generic (PLEG): container finished" podID="79bc82ee-1165-4e1c-a930-d41745214818" containerID="47fd3f13b8c0b262e5db84afb7bc35445928c2e3bf321b261c282e40be701482" exitCode=0 Sep 11 14:08:14.032365 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:14.031917 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" Sep 11 14:08:14.032365 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:14.031936 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" event={"ID":"79bc82ee-1165-4e1c-a930-d41745214818","Type":"ContainerDied","Data":"47fd3f13b8c0b262e5db84afb7bc35445928c2e3bf321b261c282e40be701482"} Sep 11 14:08:14.032365 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:14.031978 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw" event={"ID":"79bc82ee-1165-4e1c-a930-d41745214818","Type":"ContainerDied","Data":"958d8c76c7b380c69faa9ec44c5383e065c8a2e2a7bf1b5d6c3ccd2d083ac42a"} Sep 11 14:08:14.032365 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:14.031997 2566 scope.go:117] "RemoveContainer" containerID="47fd3f13b8c0b262e5db84afb7bc35445928c2e3bf321b261c282e40be701482" Sep 11 14:08:14.041210 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:14.041059 2566 scope.go:117] "RemoveContainer" containerID="47fd3f13b8c0b262e5db84afb7bc35445928c2e3bf321b261c282e40be701482" Sep 11 14:08:14.041437 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:08:14.041416 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47fd3f13b8c0b262e5db84afb7bc35445928c2e3bf321b261c282e40be701482\": container with ID starting with 47fd3f13b8c0b262e5db84afb7bc35445928c2e3bf321b261c282e40be701482 not found: ID does not exist" containerID="47fd3f13b8c0b262e5db84afb7bc35445928c2e3bf321b261c282e40be701482" Sep 11 14:08:14.041538 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:14.041442 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47fd3f13b8c0b262e5db84afb7bc35445928c2e3bf321b261c282e40be701482"} err="failed to get container status \"47fd3f13b8c0b262e5db84afb7bc35445928c2e3bf321b261c282e40be701482\": rpc error: code = NotFound desc = could not find container \"47fd3f13b8c0b262e5db84afb7bc35445928c2e3bf321b261c282e40be701482\": container with ID starting with 47fd3f13b8c0b262e5db84afb7bc35445928c2e3bf321b261c282e40be701482 not found: ID does not exist" Sep 11 14:08:14.063994 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:14.063972 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw"] Sep 11 14:08:14.069669 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:14.069643 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wxwhw"] Sep 11 14:08:14.097215 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:14.097188 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79bc82ee-1165-4e1c-a930-d41745214818" path="/var/lib/kubelet/pods/79bc82ee-1165-4e1c-a930-d41745214818/volumes" Sep 11 14:08:29.637613 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.637577 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx"] Sep 11 14:08:29.638125 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.637919 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bc82ee-1165-4e1c-a930-d41745214818" containerName="manager" Sep 11 14:08:29.638125 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.637930 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bc82ee-1165-4e1c-a930-d41745214818" containerName="manager" Sep 11 14:08:29.638125 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.637998 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="79bc82ee-1165-4e1c-a930-d41745214818" containerName="manager" Sep 11 14:08:29.641819 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.641791 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.645874 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.645853 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5hgfs\"" Sep 11 14:08:29.659248 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.659224 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx"] Sep 11 14:08:29.797927 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.797892 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.798151 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.797935 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.798151 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.797958 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.798151 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.798028 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.798151 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.798060 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.798151 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.798099 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.798151 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.798132 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trgfk\" (UniqueName: \"kubernetes.io/projected/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-kube-api-access-trgfk\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.798414 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.798199 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.798414 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.798243 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.898916 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.898817 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.898916 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.898861 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.898916 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.898889 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.899260 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.898940 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.899260 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.898964 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.899260 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.898986 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.899260 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.899017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trgfk\" (UniqueName: \"kubernetes.io/projected/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-kube-api-access-trgfk\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.899260 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.899073 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.899260 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.899145 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.899555 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.899350 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.899555 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.899375 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.899555 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.899442 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.899713 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.899612 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.899910 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.899887 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.901667 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.901641 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.901854 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.901817 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.907808 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.907772 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.907935 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.907887 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trgfk\" (UniqueName: \"kubernetes.io/projected/31429fc7-4a21-4e5d-9db7-0a0663bf8a71-kube-api-access-trgfk\") pod \"maas-default-gateway-openshift-default-78bb78ff78-t98fx\" (UID: \"31429fc7-4a21-4e5d-9db7-0a0663bf8a71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:29.951799 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:29.951774 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:30.239887 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:30.239854 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx"] Sep 11 14:08:30.241282 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:08:30.241250 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31429fc7_4a21_4e5d_9db7_0a0663bf8a71.slice/crio-613a46b73985d9fe21d5d0e59fb3766fcfa5783c7d6d900be1a6300bfb91245b WatchSource:0}: Error finding container 613a46b73985d9fe21d5d0e59fb3766fcfa5783c7d6d900be1a6300bfb91245b: Status 404 returned error can't find the container with id 613a46b73985d9fe21d5d0e59fb3766fcfa5783c7d6d900be1a6300bfb91245b Sep 11 14:08:30.243517 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:30.243478 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:08:30.243617 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:30.243563 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:08:30.243617 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:30.243603 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:08:31.132478 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:31.132442 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" event={"ID":"31429fc7-4a21-4e5d-9db7-0a0663bf8a71","Type":"ContainerStarted","Data":"98af7fd7f0a51942e2b9e1284e9ad96097b4fb147e6546bbfda79a1aeb28040d"} Sep 11 14:08:31.132889 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:31.132504 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" event={"ID":"31429fc7-4a21-4e5d-9db7-0a0663bf8a71","Type":"ContainerStarted","Data":"613a46b73985d9fe21d5d0e59fb3766fcfa5783c7d6d900be1a6300bfb91245b"} Sep 11 14:08:31.160859 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:31.160808 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" podStartSLOduration=2.160792148 podStartE2EDuration="2.160792148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:31.158373997 +0000 UTC m=+651.563355125" watchObservedRunningTime="2026-09-11 14:08:31.160792148 +0000 UTC m=+651.565773275" Sep 11 14:08:31.952633 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:31.952593 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:31.957460 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:31.957435 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:32.137631 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.137597 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:32.138538 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.138519 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:08:32.247208 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.247178 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v6xkn"] Sep 11 14:08:32.249639 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.249623 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" Sep 11 14:08:32.252288 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.252267 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 11 14:08:32.252395 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.252319 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fj6mk\"" Sep 11 14:08:32.260010 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.259988 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v6xkn"] Sep 11 14:08:32.342098 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.342050 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v6xkn"] Sep 11 14:08:32.422809 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.422775 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2714071f-3c34-4c3e-8583-9e3bda2fd374-config-file\") pod \"limitador-limitador-8588749977-v6xkn\" (UID: \"2714071f-3c34-4c3e-8583-9e3bda2fd374\") " pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" Sep 11 14:08:32.422994 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.422864 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqbmb\" (UniqueName: \"kubernetes.io/projected/2714071f-3c34-4c3e-8583-9e3bda2fd374-kube-api-access-lqbmb\") pod \"limitador-limitador-8588749977-v6xkn\" (UID: \"2714071f-3c34-4c3e-8583-9e3bda2fd374\") " pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" Sep 11 14:08:32.523634 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.523538 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2714071f-3c34-4c3e-8583-9e3bda2fd374-config-file\") pod \"limitador-limitador-8588749977-v6xkn\" (UID: \"2714071f-3c34-4c3e-8583-9e3bda2fd374\") " pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" Sep 11 14:08:32.523634 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.523621 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqbmb\" (UniqueName: \"kubernetes.io/projected/2714071f-3c34-4c3e-8583-9e3bda2fd374-kube-api-access-lqbmb\") pod \"limitador-limitador-8588749977-v6xkn\" (UID: \"2714071f-3c34-4c3e-8583-9e3bda2fd374\") " pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" Sep 11 14:08:32.524197 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.524178 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2714071f-3c34-4c3e-8583-9e3bda2fd374-config-file\") pod \"limitador-limitador-8588749977-v6xkn\" (UID: \"2714071f-3c34-4c3e-8583-9e3bda2fd374\") " pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" Sep 11 14:08:32.534213 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.534188 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqbmb\" (UniqueName: \"kubernetes.io/projected/2714071f-3c34-4c3e-8583-9e3bda2fd374-kube-api-access-lqbmb\") pod \"limitador-limitador-8588749977-v6xkn\" (UID: \"2714071f-3c34-4c3e-8583-9e3bda2fd374\") " pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" Sep 11 14:08:32.561388 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.561360 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" Sep 11 14:08:32.838974 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.838946 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v6xkn"] Sep 11 14:08:32.839886 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:08:32.839850 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2714071f_3c34_4c3e_8583_9e3bda2fd374.slice/crio-8098765fcfb848361fa24119a5ef48702f23c227c34ac76a9ec114f69c715165 WatchSource:0}: Error finding container 8098765fcfb848361fa24119a5ef48702f23c227c34ac76a9ec114f69c715165: Status 404 returned error can't find the container with id 8098765fcfb848361fa24119a5ef48702f23c227c34ac76a9ec114f69c715165 Sep 11 14:08:32.842198 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:32.842178 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:08:33.060663 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:33.060621 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n5zcx"] Sep 11 14:08:33.063918 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:33.063892 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n5zcx" Sep 11 14:08:33.066344 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:33.066324 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-skzf8\"" Sep 11 14:08:33.079098 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:33.079056 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n5zcx"] Sep 11 14:08:33.130463 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:33.130330 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64r42\" (UniqueName: \"kubernetes.io/projected/1d33f245-e029-46b4-b54c-823edc9a6df2-kube-api-access-64r42\") pod \"authorino-7bcfd8f9cd-n5zcx\" (UID: \"1d33f245-e029-46b4-b54c-823edc9a6df2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-n5zcx" Sep 11 14:08:33.144627 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:33.144593 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" event={"ID":"2714071f-3c34-4c3e-8583-9e3bda2fd374","Type":"ContainerStarted","Data":"8098765fcfb848361fa24119a5ef48702f23c227c34ac76a9ec114f69c715165"} Sep 11 14:08:33.231739 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:33.231699 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r42\" (UniqueName: \"kubernetes.io/projected/1d33f245-e029-46b4-b54c-823edc9a6df2-kube-api-access-64r42\") pod \"authorino-7bcfd8f9cd-n5zcx\" (UID: \"1d33f245-e029-46b4-b54c-823edc9a6df2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-n5zcx" Sep 11 14:08:33.244572 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:33.244534 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64r42\" (UniqueName: \"kubernetes.io/projected/1d33f245-e029-46b4-b54c-823edc9a6df2-kube-api-access-64r42\") pod \"authorino-7bcfd8f9cd-n5zcx\" (UID: \"1d33f245-e029-46b4-b54c-823edc9a6df2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-n5zcx" Sep 11 14:08:33.372907 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:33.372862 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n5zcx" Sep 11 14:08:33.643407 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:33.643379 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n5zcx"] Sep 11 14:08:33.644665 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:08:33.644642 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d33f245_e029_46b4_b54c_823edc9a6df2.slice/crio-b8e1542daee1a0a79da0438d24bf15831eb12a3399593c9d82d342d14d08ae0a WatchSource:0}: Error finding container b8e1542daee1a0a79da0438d24bf15831eb12a3399593c9d82d342d14d08ae0a: Status 404 returned error can't find the container with id b8e1542daee1a0a79da0438d24bf15831eb12a3399593c9d82d342d14d08ae0a Sep 11 14:08:34.152572 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:34.152545 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" event={"ID":"2714071f-3c34-4c3e-8583-9e3bda2fd374","Type":"ContainerStarted","Data":"675050a9b2542bf7e7d85fb4797d77d0045402c5435fd65e4f9514e50f5412e6"} Sep 11 14:08:34.152935 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:34.152696 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" Sep 11 14:08:34.154050 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:34.154030 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n5zcx" event={"ID":"1d33f245-e029-46b4-b54c-823edc9a6df2","Type":"ContainerStarted","Data":"b8e1542daee1a0a79da0438d24bf15831eb12a3399593c9d82d342d14d08ae0a"} Sep 11 14:08:34.173742 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:34.173695 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" podStartSLOduration=0.925868815 podStartE2EDuration="2.173678506s" totalImagesPullingTime="1.247809691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:34.171499196 +0000 UTC m=+654.576480325" watchObservedRunningTime="2026-09-11 14:08:34.173678506 +0000 UTC m=+654.578659680" Sep 11 14:08:36.160341 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:36.160308 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n5zcx"] Sep 11 14:08:36.172146 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:36.172076 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n5zcx" event={"ID":"1d33f245-e029-46b4-b54c-823edc9a6df2","Type":"ContainerStarted","Data":"93121ee903c930ee84fcf55affa6c4908a7b4173e0d12106f45244de7897f3b1"} Sep 11 14:08:36.218351 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:36.218299 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-n5zcx" podStartSLOduration=1.602072938 podStartE2EDuration="3.218282287s" totalImagesPullingTime="1.616209349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:36.217528629 +0000 UTC m=+656.622509756" watchObservedRunningTime="2026-09-11 14:08:36.218282287 +0000 UTC m=+656.623263414" Sep 11 14:08:37.177620 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:37.177585 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-n5zcx" podUID="1d33f245-e029-46b4-b54c-823edc9a6df2" containerName="authorino" containerID="cri-o://93121ee903c930ee84fcf55affa6c4908a7b4173e0d12106f45244de7897f3b1" gracePeriod=30 Sep 11 14:08:37.421666 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:37.421642 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n5zcx" Sep 11 14:08:37.467565 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:37.467460 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64r42\" (UniqueName: \"kubernetes.io/projected/1d33f245-e029-46b4-b54c-823edc9a6df2-kube-api-access-64r42\") pod \"1d33f245-e029-46b4-b54c-823edc9a6df2\" (UID: \"1d33f245-e029-46b4-b54c-823edc9a6df2\") " Sep 11 14:08:37.469774 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:37.469748 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d33f245-e029-46b4-b54c-823edc9a6df2-kube-api-access-64r42" (OuterVolumeSpecName: "kube-api-access-64r42") pod "1d33f245-e029-46b4-b54c-823edc9a6df2" (UID: "1d33f245-e029-46b4-b54c-823edc9a6df2"). InnerVolumeSpecName "kube-api-access-64r42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:08:37.568440 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:37.568398 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64r42\" (UniqueName: \"kubernetes.io/projected/1d33f245-e029-46b4-b54c-823edc9a6df2-kube-api-access-64r42\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:08:38.183957 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:38.183924 2566 generic.go:356] "Generic (PLEG): container finished" podID="1d33f245-e029-46b4-b54c-823edc9a6df2" containerID="93121ee903c930ee84fcf55affa6c4908a7b4173e0d12106f45244de7897f3b1" exitCode=0 Sep 11 14:08:38.184448 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:38.183969 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n5zcx" Sep 11 14:08:38.184448 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:38.183984 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n5zcx" event={"ID":"1d33f245-e029-46b4-b54c-823edc9a6df2","Type":"ContainerDied","Data":"93121ee903c930ee84fcf55affa6c4908a7b4173e0d12106f45244de7897f3b1"} Sep 11 14:08:38.184448 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:38.184033 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n5zcx" event={"ID":"1d33f245-e029-46b4-b54c-823edc9a6df2","Type":"ContainerDied","Data":"b8e1542daee1a0a79da0438d24bf15831eb12a3399593c9d82d342d14d08ae0a"} Sep 11 14:08:38.184448 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:38.184056 2566 scope.go:117] "RemoveContainer" containerID="93121ee903c930ee84fcf55affa6c4908a7b4173e0d12106f45244de7897f3b1" Sep 11 14:08:38.193106 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:38.193062 2566 scope.go:117] "RemoveContainer" containerID="93121ee903c930ee84fcf55affa6c4908a7b4173e0d12106f45244de7897f3b1" Sep 11 14:08:38.193406 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:08:38.193388 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93121ee903c930ee84fcf55affa6c4908a7b4173e0d12106f45244de7897f3b1\": container with ID starting with 93121ee903c930ee84fcf55affa6c4908a7b4173e0d12106f45244de7897f3b1 not found: ID does not exist" containerID="93121ee903c930ee84fcf55affa6c4908a7b4173e0d12106f45244de7897f3b1" Sep 11 14:08:38.193456 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:38.193415 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93121ee903c930ee84fcf55affa6c4908a7b4173e0d12106f45244de7897f3b1"} err="failed to get container status \"93121ee903c930ee84fcf55affa6c4908a7b4173e0d12106f45244de7897f3b1\": rpc error: code = NotFound desc = could not find container \"93121ee903c930ee84fcf55affa6c4908a7b4173e0d12106f45244de7897f3b1\": container with ID starting with 93121ee903c930ee84fcf55affa6c4908a7b4173e0d12106f45244de7897f3b1 not found: ID does not exist" Sep 11 14:08:38.206239 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:38.206207 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n5zcx"] Sep 11 14:08:38.215887 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:38.215862 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n5zcx"] Sep 11 14:08:40.097873 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:40.097849 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d33f245-e029-46b4-b54c-823edc9a6df2" path="/var/lib/kubelet/pods/1d33f245-e029-46b4-b54c-823edc9a6df2/volumes" Sep 11 14:08:45.162814 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:45.162785 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" Sep 11 14:08:46.208413 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:46.208378 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v6xkn"] Sep 11 14:08:46.208958 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:46.208586 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" podUID="2714071f-3c34-4c3e-8583-9e3bda2fd374" containerName="limitador" containerID="cri-o://675050a9b2542bf7e7d85fb4797d77d0045402c5435fd65e4f9514e50f5412e6" gracePeriod=30 Sep 11 14:08:46.762158 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:46.762137 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" Sep 11 14:08:46.859780 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:46.859695 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqbmb\" (UniqueName: \"kubernetes.io/projected/2714071f-3c34-4c3e-8583-9e3bda2fd374-kube-api-access-lqbmb\") pod \"2714071f-3c34-4c3e-8583-9e3bda2fd374\" (UID: \"2714071f-3c34-4c3e-8583-9e3bda2fd374\") " Sep 11 14:08:46.859780 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:46.859739 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2714071f-3c34-4c3e-8583-9e3bda2fd374-config-file\") pod \"2714071f-3c34-4c3e-8583-9e3bda2fd374\" (UID: \"2714071f-3c34-4c3e-8583-9e3bda2fd374\") " Sep 11 14:08:46.860133 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:46.860078 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2714071f-3c34-4c3e-8583-9e3bda2fd374-config-file" (OuterVolumeSpecName: "config-file") pod "2714071f-3c34-4c3e-8583-9e3bda2fd374" (UID: "2714071f-3c34-4c3e-8583-9e3bda2fd374"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:08:46.861905 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:46.861881 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2714071f-3c34-4c3e-8583-9e3bda2fd374-kube-api-access-lqbmb" (OuterVolumeSpecName: "kube-api-access-lqbmb") pod "2714071f-3c34-4c3e-8583-9e3bda2fd374" (UID: "2714071f-3c34-4c3e-8583-9e3bda2fd374"). InnerVolumeSpecName "kube-api-access-lqbmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:08:46.960370 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:46.960332 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqbmb\" (UniqueName: \"kubernetes.io/projected/2714071f-3c34-4c3e-8583-9e3bda2fd374-kube-api-access-lqbmb\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:08:46.960370 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:46.960363 2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2714071f-3c34-4c3e-8583-9e3bda2fd374-config-file\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:08:47.235122 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:47.235075 2566 generic.go:356] "Generic (PLEG): container finished" podID="2714071f-3c34-4c3e-8583-9e3bda2fd374" containerID="675050a9b2542bf7e7d85fb4797d77d0045402c5435fd65e4f9514e50f5412e6" exitCode=0 Sep 11 14:08:47.235550 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:47.235120 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" event={"ID":"2714071f-3c34-4c3e-8583-9e3bda2fd374","Type":"ContainerDied","Data":"675050a9b2542bf7e7d85fb4797d77d0045402c5435fd65e4f9514e50f5412e6"} Sep 11 14:08:47.235550 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:47.235145 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" Sep 11 14:08:47.235550 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:47.235181 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v6xkn" event={"ID":"2714071f-3c34-4c3e-8583-9e3bda2fd374","Type":"ContainerDied","Data":"8098765fcfb848361fa24119a5ef48702f23c227c34ac76a9ec114f69c715165"} Sep 11 14:08:47.235550 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:47.235207 2566 scope.go:117] "RemoveContainer" containerID="675050a9b2542bf7e7d85fb4797d77d0045402c5435fd65e4f9514e50f5412e6" Sep 11 14:08:47.245247 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:47.245230 2566 scope.go:117] "RemoveContainer" containerID="675050a9b2542bf7e7d85fb4797d77d0045402c5435fd65e4f9514e50f5412e6" Sep 11 14:08:47.245541 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:08:47.245520 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"675050a9b2542bf7e7d85fb4797d77d0045402c5435fd65e4f9514e50f5412e6\": container with ID starting with 675050a9b2542bf7e7d85fb4797d77d0045402c5435fd65e4f9514e50f5412e6 not found: ID does not exist" containerID="675050a9b2542bf7e7d85fb4797d77d0045402c5435fd65e4f9514e50f5412e6" Sep 11 14:08:47.245611 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:47.245548 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"675050a9b2542bf7e7d85fb4797d77d0045402c5435fd65e4f9514e50f5412e6"} err="failed to get container status \"675050a9b2542bf7e7d85fb4797d77d0045402c5435fd65e4f9514e50f5412e6\": rpc error: code = NotFound desc = could not find container \"675050a9b2542bf7e7d85fb4797d77d0045402c5435fd65e4f9514e50f5412e6\": container with ID starting with 675050a9b2542bf7e7d85fb4797d77d0045402c5435fd65e4f9514e50f5412e6 not found: ID does not exist" Sep 11 14:08:47.283986 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:47.283950 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v6xkn"] Sep 11 14:08:47.303974 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:47.303942 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v6xkn"] Sep 11 14:08:48.097803 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:48.097770 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2714071f-3c34-4c3e-8583-9e3bda2fd374" path="/var/lib/kubelet/pods/2714071f-3c34-4c3e-8583-9e3bda2fd374/volumes" Sep 11 14:08:48.938478 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:48.938444 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6"] Sep 11 14:08:48.938852 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:48.938810 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2714071f-3c34-4c3e-8583-9e3bda2fd374" containerName="limitador" Sep 11 14:08:48.938852 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:48.938822 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2714071f-3c34-4c3e-8583-9e3bda2fd374" containerName="limitador" Sep 11 14:08:48.938852 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:48.938834 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d33f245-e029-46b4-b54c-823edc9a6df2" containerName="authorino" Sep 11 14:08:48.938852 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:48.938840 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d33f245-e029-46b4-b54c-823edc9a6df2" containerName="authorino" Sep 11 14:08:48.939005 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:48.938906 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d33f245-e029-46b4-b54c-823edc9a6df2" containerName="authorino" Sep 11 14:08:48.939005 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:48.938918 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2714071f-3c34-4c3e-8583-9e3bda2fd374" containerName="limitador" Sep 11 14:08:48.943275 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:48.943256 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6" Sep 11 14:08:48.946059 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:48.946039 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-w8l4c\"" Sep 11 14:08:48.946326 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:48.946308 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:08:48.946906 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:48.946890 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 14:08:48.946998 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:48.946901 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 11 14:08:48.960142 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:48.960077 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6"] Sep 11 14:08:49.078430 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:49.078390 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tswfm\" (UniqueName: \"kubernetes.io/projected/90862da7-61de-4138-a7fb-7c2787d43ae8-kube-api-access-tswfm\") pod \"postgres-69fb4594cf-s8dp6\" (UID: \"90862da7-61de-4138-a7fb-7c2787d43ae8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6" Sep 11 14:08:49.078614 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:49.078449 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/90862da7-61de-4138-a7fb-7c2787d43ae8-data\") pod \"postgres-69fb4594cf-s8dp6\" (UID: \"90862da7-61de-4138-a7fb-7c2787d43ae8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6" Sep 11 14:08:49.178973 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:49.178934 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tswfm\" (UniqueName: \"kubernetes.io/projected/90862da7-61de-4138-a7fb-7c2787d43ae8-kube-api-access-tswfm\") pod \"postgres-69fb4594cf-s8dp6\" (UID: \"90862da7-61de-4138-a7fb-7c2787d43ae8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6" Sep 11 14:08:49.179179 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:49.178992 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/90862da7-61de-4138-a7fb-7c2787d43ae8-data\") pod \"postgres-69fb4594cf-s8dp6\" (UID: \"90862da7-61de-4138-a7fb-7c2787d43ae8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6" Sep 11 14:08:49.179335 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:49.179315 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/90862da7-61de-4138-a7fb-7c2787d43ae8-data\") pod \"postgres-69fb4594cf-s8dp6\" (UID: \"90862da7-61de-4138-a7fb-7c2787d43ae8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6" Sep 11 14:08:49.193137 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:49.193059 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tswfm\" (UniqueName: \"kubernetes.io/projected/90862da7-61de-4138-a7fb-7c2787d43ae8-kube-api-access-tswfm\") pod \"postgres-69fb4594cf-s8dp6\" (UID: \"90862da7-61de-4138-a7fb-7c2787d43ae8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6" Sep 11 14:08:49.253839 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:49.253812 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6" Sep 11 14:08:49.549626 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:49.549601 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6"] Sep 11 14:08:49.550933 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:08:49.550905 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90862da7_61de_4138_a7fb_7c2787d43ae8.slice/crio-86e421beee4ebd63ab80e4e8352a0c87eaa460cc09dbace9343fe3c87ba1d231 WatchSource:0}: Error finding container 86e421beee4ebd63ab80e4e8352a0c87eaa460cc09dbace9343fe3c87ba1d231: Status 404 returned error can't find the container with id 86e421beee4ebd63ab80e4e8352a0c87eaa460cc09dbace9343fe3c87ba1d231 Sep 11 14:08:50.258901 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:50.258862 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6" event={"ID":"90862da7-61de-4138-a7fb-7c2787d43ae8","Type":"ContainerStarted","Data":"86e421beee4ebd63ab80e4e8352a0c87eaa460cc09dbace9343fe3c87ba1d231"} Sep 11 14:08:55.293498 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:55.293458 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6" event={"ID":"90862da7-61de-4138-a7fb-7c2787d43ae8","Type":"ContainerStarted","Data":"887e3423fa01808148db2e06dfe5d17dacd4aa684d0f725da1649ed399fe7cd9"} Sep 11 14:08:55.293921 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:55.293535 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6" Sep 11 14:08:55.331979 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:08:55.331924 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6" podStartSLOduration=2.596131399 podStartE2EDuration="7.331907643s" totalImagesPullingTime="4.735776244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:55.330172364 +0000 UTC m=+675.735153492" watchObservedRunningTime="2026-09-11 14:08:55.331907643 +0000 UTC m=+675.736888772" Sep 11 14:09:01.327948 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:01.327920 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s8dp6" Sep 11 14:09:01.890936 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:01.890907 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-m8vqc"] Sep 11 14:09:01.896928 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:01.896907 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-m8vqc" Sep 11 14:09:01.899581 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:01.899561 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-skzf8\"" Sep 11 14:09:01.904056 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:01.904033 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-m8vqc"] Sep 11 14:09:01.940657 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:01.940629 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-m8vqc"] Sep 11 14:09:01.940890 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:09:01.940866 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5g5b5], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-m8vqc" podUID="8bca5907-2611-4712-aa50-69214ee9fb18" Sep 11 14:09:01.974530 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:01.974500 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pjjms"] Sep 11 14:09:01.978597 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:01.978575 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pjjms" Sep 11 14:09:01.981274 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:01.981257 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 11 14:09:01.989393 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:01.989371 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pjjms"] Sep 11 14:09:02.002080 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.002053 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g5b5\" (UniqueName: \"kubernetes.io/projected/8bca5907-2611-4712-aa50-69214ee9fb18-kube-api-access-5g5b5\") pod \"authorino-557f876f69-m8vqc\" (UID: \"8bca5907-2611-4712-aa50-69214ee9fb18\") " pod="kuadrant-system/authorino-557f876f69-m8vqc" Sep 11 14:09:02.102666 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.102636 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41bc2354-298d-4323-a7be-f8b0f6361c38-tls-cert\") pod \"authorino-6d4dfc649f-pjjms\" (UID: \"41bc2354-298d-4323-a7be-f8b0f6361c38\") " pod="kuadrant-system/authorino-6d4dfc649f-pjjms" Sep 11 14:09:02.102860 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.102696 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g5b5\" (UniqueName: \"kubernetes.io/projected/8bca5907-2611-4712-aa50-69214ee9fb18-kube-api-access-5g5b5\") pod \"authorino-557f876f69-m8vqc\" (UID: \"8bca5907-2611-4712-aa50-69214ee9fb18\") " pod="kuadrant-system/authorino-557f876f69-m8vqc" Sep 11 14:09:02.102860 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.102815 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2knn\" (UniqueName: \"kubernetes.io/projected/41bc2354-298d-4323-a7be-f8b0f6361c38-kube-api-access-v2knn\") pod \"authorino-6d4dfc649f-pjjms\" (UID: \"41bc2354-298d-4323-a7be-f8b0f6361c38\") " pod="kuadrant-system/authorino-6d4dfc649f-pjjms" Sep 11 14:09:02.112069 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.112036 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g5b5\" (UniqueName: \"kubernetes.io/projected/8bca5907-2611-4712-aa50-69214ee9fb18-kube-api-access-5g5b5\") pod \"authorino-557f876f69-m8vqc\" (UID: \"8bca5907-2611-4712-aa50-69214ee9fb18\") " pod="kuadrant-system/authorino-557f876f69-m8vqc" Sep 11 14:09:02.144714 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.144640 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pjjms"] Sep 11 14:09:02.144947 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:09:02.144926 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-v2knn tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-pjjms" podUID="41bc2354-298d-4323-a7be-f8b0f6361c38" Sep 11 14:09:02.185709 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.185674 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-775884f7c9-zgk92"] Sep 11 14:09:02.189736 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.189707 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775884f7c9-zgk92" Sep 11 14:09:02.197954 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.197928 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-775884f7c9-zgk92"] Sep 11 14:09:02.203443 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.203415 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2knn\" (UniqueName: \"kubernetes.io/projected/41bc2354-298d-4323-a7be-f8b0f6361c38-kube-api-access-v2knn\") pod \"authorino-6d4dfc649f-pjjms\" (UID: \"41bc2354-298d-4323-a7be-f8b0f6361c38\") " pod="kuadrant-system/authorino-6d4dfc649f-pjjms" Sep 11 14:09:02.203557 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.203536 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41bc2354-298d-4323-a7be-f8b0f6361c38-tls-cert\") pod \"authorino-6d4dfc649f-pjjms\" (UID: \"41bc2354-298d-4323-a7be-f8b0f6361c38\") " pod="kuadrant-system/authorino-6d4dfc649f-pjjms" Sep 11 14:09:02.206423 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.206399 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41bc2354-298d-4323-a7be-f8b0f6361c38-tls-cert\") pod \"authorino-6d4dfc649f-pjjms\" (UID: \"41bc2354-298d-4323-a7be-f8b0f6361c38\") " pod="kuadrant-system/authorino-6d4dfc649f-pjjms" Sep 11 14:09:02.213399 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.213374 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2knn\" (UniqueName: \"kubernetes.io/projected/41bc2354-298d-4323-a7be-f8b0f6361c38-kube-api-access-v2knn\") pod \"authorino-6d4dfc649f-pjjms\" (UID: \"41bc2354-298d-4323-a7be-f8b0f6361c38\") " pod="kuadrant-system/authorino-6d4dfc649f-pjjms" Sep 11 14:09:02.304363 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.304329 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7jg6\" (UniqueName: \"kubernetes.io/projected/84ffb829-1224-4aa6-af3a-660da4a9217a-kube-api-access-p7jg6\") pod \"authorino-775884f7c9-zgk92\" (UID: \"84ffb829-1224-4aa6-af3a-660da4a9217a\") " pod="kuadrant-system/authorino-775884f7c9-zgk92" Sep 11 14:09:02.304533 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.304377 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84ffb829-1224-4aa6-af3a-660da4a9217a-tls-cert\") pod \"authorino-775884f7c9-zgk92\" (UID: \"84ffb829-1224-4aa6-af3a-660da4a9217a\") " pod="kuadrant-system/authorino-775884f7c9-zgk92" Sep 11 14:09:02.332038 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.332019 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-m8vqc" Sep 11 14:09:02.332415 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.332019 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pjjms" Sep 11 14:09:02.339850 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.339829 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-m8vqc" Sep 11 14:09:02.342699 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.342683 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pjjms" Sep 11 14:09:02.405365 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.405275 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7jg6\" (UniqueName: \"kubernetes.io/projected/84ffb829-1224-4aa6-af3a-660da4a9217a-kube-api-access-p7jg6\") pod \"authorino-775884f7c9-zgk92\" (UID: \"84ffb829-1224-4aa6-af3a-660da4a9217a\") " pod="kuadrant-system/authorino-775884f7c9-zgk92" Sep 11 14:09:02.405365 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.405317 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84ffb829-1224-4aa6-af3a-660da4a9217a-tls-cert\") pod \"authorino-775884f7c9-zgk92\" (UID: \"84ffb829-1224-4aa6-af3a-660da4a9217a\") " pod="kuadrant-system/authorino-775884f7c9-zgk92" Sep 11 14:09:02.407870 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.407848 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84ffb829-1224-4aa6-af3a-660da4a9217a-tls-cert\") pod \"authorino-775884f7c9-zgk92\" (UID: \"84ffb829-1224-4aa6-af3a-660da4a9217a\") " pod="kuadrant-system/authorino-775884f7c9-zgk92" Sep 11 14:09:02.415157 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.415133 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7jg6\" (UniqueName: \"kubernetes.io/projected/84ffb829-1224-4aa6-af3a-660da4a9217a-kube-api-access-p7jg6\") pod \"authorino-775884f7c9-zgk92\" (UID: \"84ffb829-1224-4aa6-af3a-660da4a9217a\") " pod="kuadrant-system/authorino-775884f7c9-zgk92" Sep 11 14:09:02.500202 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.500152 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775884f7c9-zgk92" Sep 11 14:09:02.505750 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.505729 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2knn\" (UniqueName: \"kubernetes.io/projected/41bc2354-298d-4323-a7be-f8b0f6361c38-kube-api-access-v2knn\") pod \"41bc2354-298d-4323-a7be-f8b0f6361c38\" (UID: \"41bc2354-298d-4323-a7be-f8b0f6361c38\") " Sep 11 14:09:02.505864 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.505806 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g5b5\" (UniqueName: \"kubernetes.io/projected/8bca5907-2611-4712-aa50-69214ee9fb18-kube-api-access-5g5b5\") pod \"8bca5907-2611-4712-aa50-69214ee9fb18\" (UID: \"8bca5907-2611-4712-aa50-69214ee9fb18\") " Sep 11 14:09:02.505934 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.505920 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41bc2354-298d-4323-a7be-f8b0f6361c38-tls-cert\") pod \"41bc2354-298d-4323-a7be-f8b0f6361c38\" (UID: \"41bc2354-298d-4323-a7be-f8b0f6361c38\") " Sep 11 14:09:02.508079 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.508047 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bca5907-2611-4712-aa50-69214ee9fb18-kube-api-access-5g5b5" (OuterVolumeSpecName: "kube-api-access-5g5b5") pod "8bca5907-2611-4712-aa50-69214ee9fb18" (UID: "8bca5907-2611-4712-aa50-69214ee9fb18"). InnerVolumeSpecName "kube-api-access-5g5b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:09:02.508218 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.508101 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bc2354-298d-4323-a7be-f8b0f6361c38-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "41bc2354-298d-4323-a7be-f8b0f6361c38" (UID: "41bc2354-298d-4323-a7be-f8b0f6361c38"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:09:02.508218 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.508152 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41bc2354-298d-4323-a7be-f8b0f6361c38-kube-api-access-v2knn" (OuterVolumeSpecName: "kube-api-access-v2knn") pod "41bc2354-298d-4323-a7be-f8b0f6361c38" (UID: "41bc2354-298d-4323-a7be-f8b0f6361c38"). InnerVolumeSpecName "kube-api-access-v2knn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:09:02.607351 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.607317 2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/41bc2354-298d-4323-a7be-f8b0f6361c38-tls-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:09:02.607351 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.607344 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2knn\" (UniqueName: \"kubernetes.io/projected/41bc2354-298d-4323-a7be-f8b0f6361c38-kube-api-access-v2knn\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:09:02.607351 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.607356 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g5b5\" (UniqueName: \"kubernetes.io/projected/8bca5907-2611-4712-aa50-69214ee9fb18-kube-api-access-5g5b5\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:09:02.772197 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:02.772168 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-775884f7c9-zgk92"] Sep 11 14:09:02.773615 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:09:02.773588 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84ffb829_1224_4aa6_af3a_660da4a9217a.slice/crio-3554ba69b8afcbb97962114d44a2cb443384bf7470927e911786975b57a432de WatchSource:0}: Error finding container 3554ba69b8afcbb97962114d44a2cb443384bf7470927e911786975b57a432de: Status 404 returned error can't find the container with id 3554ba69b8afcbb97962114d44a2cb443384bf7470927e911786975b57a432de Sep 11 14:09:03.339986 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:03.339946 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-pjjms" Sep 11 14:09:03.339986 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:03.339959 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775884f7c9-zgk92" event={"ID":"84ffb829-1224-4aa6-af3a-660da4a9217a","Type":"ContainerStarted","Data":"3554ba69b8afcbb97962114d44a2cb443384bf7470927e911786975b57a432de"} Sep 11 14:09:03.339986 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:03.339990 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-m8vqc" Sep 11 14:09:03.377236 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:03.377164 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-m8vqc"] Sep 11 14:09:03.379966 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:03.379942 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-m8vqc"] Sep 11 14:09:03.414417 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:03.414390 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pjjms"] Sep 11 14:09:03.421009 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:03.420985 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-pjjms"] Sep 11 14:09:04.098612 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:04.098576 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41bc2354-298d-4323-a7be-f8b0f6361c38" path="/var/lib/kubelet/pods/41bc2354-298d-4323-a7be-f8b0f6361c38/volumes" Sep 11 14:09:04.098803 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:04.098791 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bca5907-2611-4712-aa50-69214ee9fb18" path="/var/lib/kubelet/pods/8bca5907-2611-4712-aa50-69214ee9fb18/volumes" Sep 11 14:09:04.347435 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:04.347398 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775884f7c9-zgk92" event={"ID":"84ffb829-1224-4aa6-af3a-660da4a9217a","Type":"ContainerStarted","Data":"743d2b3d5942d96b093ebf51ea79ae5b30e5bb072adc9c902918bc5cfe734817"} Sep 11 14:09:04.371521 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:04.371410 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-775884f7c9-zgk92" podStartSLOduration=1.745445751 podStartE2EDuration="2.371392162s" totalImagesPullingTime="625.946411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:09:04.370573878 +0000 UTC m=+684.775555008" watchObservedRunningTime="2026-09-11 14:09:04.371392162 +0000 UTC m=+684.776373289" Sep 11 14:09:47.143263 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:47.143230 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-rjfcb"] Sep 11 14:09:47.146431 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:47.146412 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-rjfcb" Sep 11 14:09:47.149418 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:09:47.149387 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-dockercfg-g6k67\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-g6k67\"" type="*v1.Secret" Sep 11 14:09:47.149418 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:09:47.149389 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-processing-5b54b7fbb7-rjfcb\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="417e7a61-fc2b-4125-97cf-53ef51e93982" pod="openshift-ingress/payload-processing-5b54b7fbb7-rjfcb" Sep 11 14:09:47.150049 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:09:47.150027 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"payload-processing-plugins\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" type="*v1.ConfigMap" Sep 11 14:09:47.174364 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:47.174326 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-rjfcb"] Sep 11 14:09:47.285982 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:47.285943 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm9jn\" (UniqueName: \"kubernetes.io/projected/417e7a61-fc2b-4125-97cf-53ef51e93982-kube-api-access-rm9jn\") pod \"payload-processing-5b54b7fbb7-rjfcb\" (UID: \"417e7a61-fc2b-4125-97cf-53ef51e93982\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-rjfcb" Sep 11 14:09:47.286206 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:47.285997 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/417e7a61-fc2b-4125-97cf-53ef51e93982-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-rjfcb\" (UID: \"417e7a61-fc2b-4125-97cf-53ef51e93982\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-rjfcb" Sep 11 14:09:47.386556 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:47.386515 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm9jn\" (UniqueName: \"kubernetes.io/projected/417e7a61-fc2b-4125-97cf-53ef51e93982-kube-api-access-rm9jn\") pod \"payload-processing-5b54b7fbb7-rjfcb\" (UID: \"417e7a61-fc2b-4125-97cf-53ef51e93982\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-rjfcb" Sep 11 14:09:47.386729 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:47.386568 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/417e7a61-fc2b-4125-97cf-53ef51e93982-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-rjfcb\" (UID: \"417e7a61-fc2b-4125-97cf-53ef51e93982\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-rjfcb" Sep 11 14:09:47.419387 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:47.419252 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm9jn\" (UniqueName: \"kubernetes.io/projected/417e7a61-fc2b-4125-97cf-53ef51e93982-kube-api-access-rm9jn\") pod \"payload-processing-5b54b7fbb7-rjfcb\" (UID: \"417e7a61-fc2b-4125-97cf-53ef51e93982\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-rjfcb" Sep 11 14:09:48.285745 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:48.285712 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-g6k67\"" Sep 11 14:09:48.300764 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:48.300732 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 14:09:48.308504 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:48.308472 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/417e7a61-fc2b-4125-97cf-53ef51e93982-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-rjfcb\" (UID: \"417e7a61-fc2b-4125-97cf-53ef51e93982\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-rjfcb" Sep 11 14:09:48.357708 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:48.357661 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-rjfcb" Sep 11 14:09:48.667857 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:48.667827 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-rjfcb"] Sep 11 14:09:48.668964 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:09:48.668939 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417e7a61_fc2b_4125_97cf_53ef51e93982.slice/crio-8349fc393b29cefad66138eb64b5e1a9a89970ab81f21e6aa9692bea736d8a91 WatchSource:0}: Error finding container 8349fc393b29cefad66138eb64b5e1a9a89970ab81f21e6aa9692bea736d8a91: Status 404 returned error can't find the container with id 8349fc393b29cefad66138eb64b5e1a9a89970ab81f21e6aa9692bea736d8a91 Sep 11 14:09:49.625109 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:49.625054 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-rjfcb" event={"ID":"417e7a61-fc2b-4125-97cf-53ef51e93982","Type":"ContainerStarted","Data":"8349fc393b29cefad66138eb64b5e1a9a89970ab81f21e6aa9692bea736d8a91"} Sep 11 14:09:50.633401 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:50.633304 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-rjfcb" event={"ID":"417e7a61-fc2b-4125-97cf-53ef51e93982","Type":"ContainerStarted","Data":"8bfe8424a972804febfb0378dba8d05e9ff17d16d5d1af392073a675a358dece"} Sep 11 14:09:50.633899 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:50.633880 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-rjfcb" Sep 11 14:09:50.667548 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:50.667494 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-rjfcb" podStartSLOduration=1.978715165 podStartE2EDuration="3.66747702s" totalImagesPullingTime="1.688761855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:09:50.666292719 +0000 UTC m=+731.071273847" watchObservedRunningTime="2026-09-11 14:09:50.66747702 +0000 UTC m=+731.072458149" Sep 11 14:09:56.671721 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:56.671683 2566 generic.go:356] "Generic (PLEG): container finished" podID="31429fc7-4a21-4e5d-9db7-0a0663bf8a71" containerID="98af7fd7f0a51942e2b9e1284e9ad96097b4fb147e6546bbfda79a1aeb28040d" exitCode=0 Sep 11 14:09:56.672185 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:56.671752 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" event={"ID":"31429fc7-4a21-4e5d-9db7-0a0663bf8a71","Type":"ContainerDied","Data":"98af7fd7f0a51942e2b9e1284e9ad96097b4fb147e6546bbfda79a1aeb28040d"} Sep 11 14:09:56.672185 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:56.672166 2566 scope.go:117] "RemoveContainer" containerID="98af7fd7f0a51942e2b9e1284e9ad96097b4fb147e6546bbfda79a1aeb28040d" Sep 11 14:09:56.673307 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:56.673276 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:09:56.673386 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:56.673372 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:09:56.673429 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:56.673411 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:09:57.680126 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:57.680063 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" event={"ID":"31429fc7-4a21-4e5d-9db7-0a0663bf8a71","Type":"ContainerStarted","Data":"a8d7a2105e19c41bdbe8e2f3919fc0c8b8a17c52ddbc9dc8a131c998d2855b8d"} Sep 11 14:09:57.953043 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:57.952955 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:09:57.954212 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:57.954180 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-t98fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 11 14:09:57.954341 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:57.954233 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" podUID="31429fc7-4a21-4e5d-9db7-0a0663bf8a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 11 14:09:58.953281 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:58.953240 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-t98fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 11 14:09:58.953698 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:58.953303 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" podUID="31429fc7-4a21-4e5d-9db7-0a0663bf8a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 11 14:09:59.953101 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:59.953060 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-t98fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 11 14:09:59.953281 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:09:59.953142 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" podUID="31429fc7-4a21-4e5d-9db7-0a0663bf8a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 11 14:10:00.952704 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:00.952668 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-t98fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 11 14:10:00.953151 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:00.952733 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" podUID="31429fc7-4a21-4e5d-9db7-0a0663bf8a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 11 14:10:01.642168 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:01.642138 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-rjfcb" Sep 11 14:10:01.952372 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:01.952282 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-t98fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 11 14:10:01.952372 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:01.952342 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" podUID="31429fc7-4a21-4e5d-9db7-0a0663bf8a71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 11 14:10:02.137819 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:02.137779 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:10:02.956132 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:02.956069 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:10:02.957121 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:02.957099 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-t98fx" Sep 11 14:10:21.561995 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.561953 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n"] Sep 11 14:10:21.565150 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.565130 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.567772 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.567751 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 14:10:21.568721 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.568698 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7lnrv\"" Sep 11 14:10:21.568835 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.568720 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 11 14:10:21.568835 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.568720 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 14:10:21.576385 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.576364 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n"] Sep 11 14:10:21.611706 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.611672 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b2883f5-5b16-49be-8540-cb7d7967af6a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.611875 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.611723 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h4m4\" (UniqueName: \"kubernetes.io/projected/4b2883f5-5b16-49be-8540-cb7d7967af6a-kube-api-access-6h4m4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.611875 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.611806 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2883f5-5b16-49be-8540-cb7d7967af6a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.611875 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.611836 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2883f5-5b16-49be-8540-cb7d7967af6a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.611875 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.611860 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2883f5-5b16-49be-8540-cb7d7967af6a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.612052 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.611880 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2883f5-5b16-49be-8540-cb7d7967af6a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.713165 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.713129 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b2883f5-5b16-49be-8540-cb7d7967af6a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.713352 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.713176 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h4m4\" (UniqueName: \"kubernetes.io/projected/4b2883f5-5b16-49be-8540-cb7d7967af6a-kube-api-access-6h4m4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.713352 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.713239 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2883f5-5b16-49be-8540-cb7d7967af6a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.713352 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.713258 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2883f5-5b16-49be-8540-cb7d7967af6a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.713352 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.713283 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2883f5-5b16-49be-8540-cb7d7967af6a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.713352 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.713314 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2883f5-5b16-49be-8540-cb7d7967af6a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.713618 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.713568 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b2883f5-5b16-49be-8540-cb7d7967af6a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.713618 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.713599 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b2883f5-5b16-49be-8540-cb7d7967af6a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.713741 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.713708 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b2883f5-5b16-49be-8540-cb7d7967af6a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.715789 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.715766 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b2883f5-5b16-49be-8540-cb7d7967af6a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.716007 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.715986 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2883f5-5b16-49be-8540-cb7d7967af6a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.722382 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.722353 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h4m4\" (UniqueName: \"kubernetes.io/projected/4b2883f5-5b16-49be-8540-cb7d7967af6a-kube-api-access-6h4m4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x5m7n\" (UID: \"4b2883f5-5b16-49be-8540-cb7d7967af6a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:21.876006 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:21.875920 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:22.152018 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:22.151989 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n"] Sep 11 14:10:22.153813 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:10:22.153783 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2883f5_5b16_49be_8540_cb7d7967af6a.slice/crio-4680e5b70724964b1b53a828dabb718ab5cc7981d9d1e24f561102caeada1e19 WatchSource:0}: Error finding container 4680e5b70724964b1b53a828dabb718ab5cc7981d9d1e24f561102caeada1e19: Status 404 returned error can't find the container with id 4680e5b70724964b1b53a828dabb718ab5cc7981d9d1e24f561102caeada1e19 Sep 11 14:10:22.827440 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:22.827395 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" event={"ID":"4b2883f5-5b16-49be-8540-cb7d7967af6a","Type":"ContainerStarted","Data":"4680e5b70724964b1b53a828dabb718ab5cc7981d9d1e24f561102caeada1e19"} Sep 11 14:10:23.834536 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:23.834497 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" event={"ID":"4b2883f5-5b16-49be-8540-cb7d7967af6a","Type":"ContainerStarted","Data":"70c7bb2ba8bc5ed85636240eb0e0f673e67c16eee04ea19baa1f0690d07e80a4"} Sep 11 14:10:23.860194 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:23.860142 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" podStartSLOduration=1.7326266590000001 podStartE2EDuration="2.860124874s" totalImagesPullingTime="1.127498215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:10:23.859353649 +0000 UTC m=+764.264334777" watchObservedRunningTime="2026-09-11 14:10:23.860124874 +0000 UTC m=+764.265106001" Sep 11 14:10:31.876287 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:31.876252 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:31.876899 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:31.876321 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:31.889922 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:31.889888 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:32.898214 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:32.898183 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x5m7n" Sep 11 14:10:37.994281 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:37.994247 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl"] Sep 11 14:10:37.997228 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:37.997202 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.000270 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.000245 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 11 14:10:38.008986 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.008963 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl"] Sep 11 14:10:38.166123 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.166068 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62bcf888-fbca-4262-9906-2bc49aa00b85-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.166123 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.166129 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62bcf888-fbca-4262-9906-2bc49aa00b85-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.166356 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.166202 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjbgx\" (UniqueName: \"kubernetes.io/projected/62bcf888-fbca-4262-9906-2bc49aa00b85-kube-api-access-pjbgx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.166356 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.166233 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62bcf888-fbca-4262-9906-2bc49aa00b85-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.166356 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.166287 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62bcf888-fbca-4262-9906-2bc49aa00b85-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.166356 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.166303 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62bcf888-fbca-4262-9906-2bc49aa00b85-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.266944 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.266849 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62bcf888-fbca-4262-9906-2bc49aa00b85-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.266944 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.266889 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62bcf888-fbca-4262-9906-2bc49aa00b85-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.267225 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.266948 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62bcf888-fbca-4262-9906-2bc49aa00b85-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.267225 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.266970 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62bcf888-fbca-4262-9906-2bc49aa00b85-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.267225 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.267035 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjbgx\" (UniqueName: \"kubernetes.io/projected/62bcf888-fbca-4262-9906-2bc49aa00b85-kube-api-access-pjbgx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.267225 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.267066 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62bcf888-fbca-4262-9906-2bc49aa00b85-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.267514 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.267330 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62bcf888-fbca-4262-9906-2bc49aa00b85-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.267514 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.267373 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62bcf888-fbca-4262-9906-2bc49aa00b85-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.267514 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.267387 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62bcf888-fbca-4262-9906-2bc49aa00b85-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.269646 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.269622 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62bcf888-fbca-4262-9906-2bc49aa00b85-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.269849 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.269833 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62bcf888-fbca-4262-9906-2bc49aa00b85-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.277005 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.276981 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjbgx\" (UniqueName: \"kubernetes.io/projected/62bcf888-fbca-4262-9906-2bc49aa00b85-kube-api-access-pjbgx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sscpl\" (UID: \"62bcf888-fbca-4262-9906-2bc49aa00b85\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.308964 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.308926 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:38.582509 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.582484 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl"] Sep 11 14:10:38.584673 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:10:38.584644 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62bcf888_fbca_4262_9906_2bc49aa00b85.slice/crio-c968d7e208ed9a1ca497ad8f40513e2ece5b59806c8c3ff5427d0aef78c6e65b WatchSource:0}: Error finding container c968d7e208ed9a1ca497ad8f40513e2ece5b59806c8c3ff5427d0aef78c6e65b: Status 404 returned error can't find the container with id c968d7e208ed9a1ca497ad8f40513e2ece5b59806c8c3ff5427d0aef78c6e65b Sep 11 14:10:38.921474 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.921378 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" event={"ID":"62bcf888-fbca-4262-9906-2bc49aa00b85","Type":"ContainerStarted","Data":"26507ead65256acefa441ee2cd240c5857ce716b223d29d49f3537b7795215ba"} Sep 11 14:10:38.921474 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.921435 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" event={"ID":"62bcf888-fbca-4262-9906-2bc49aa00b85","Type":"ContainerStarted","Data":"c968d7e208ed9a1ca497ad8f40513e2ece5b59806c8c3ff5427d0aef78c6e65b"} Sep 11 14:10:38.940145 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:38.940061 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" podStartSLOduration=1.7741467 podStartE2EDuration="1.940043797s" totalImagesPullingTime="165.897097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:10:38.938847561 +0000 UTC m=+779.343828689" watchObservedRunningTime="2026-09-11 14:10:38.940043797 +0000 UTC m=+779.345024925" Sep 11 14:10:48.309880 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:48.309842 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:48.310326 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:48.309898 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:48.322207 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:48.322185 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:10:48.988464 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:10:48.988435 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sscpl" Sep 11 14:12:40.101394 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:40.101302 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:12:40.110437 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:40.110413 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:12:52.374436 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:52.374402 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5bd69597d-7sb7p"] Sep 11 14:12:52.377938 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:52.377921 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bd69597d-7sb7p" Sep 11 14:12:52.418677 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:52.418647 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bd69597d-7sb7p"] Sep 11 14:12:52.472551 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:52.472512 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8a29ab4-b886-48d2-9579-3e3a1dba2a78-tls-cert\") pod \"authorino-5bd69597d-7sb7p\" (UID: \"c8a29ab4-b886-48d2-9579-3e3a1dba2a78\") " pod="kuadrant-system/authorino-5bd69597d-7sb7p" Sep 11 14:12:52.472551 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:52.472552 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z9hc\" (UniqueName: \"kubernetes.io/projected/c8a29ab4-b886-48d2-9579-3e3a1dba2a78-kube-api-access-6z9hc\") pod \"authorino-5bd69597d-7sb7p\" (UID: \"c8a29ab4-b886-48d2-9579-3e3a1dba2a78\") " pod="kuadrant-system/authorino-5bd69597d-7sb7p" Sep 11 14:12:52.573852 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:52.573814 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8a29ab4-b886-48d2-9579-3e3a1dba2a78-tls-cert\") pod \"authorino-5bd69597d-7sb7p\" (UID: \"c8a29ab4-b886-48d2-9579-3e3a1dba2a78\") " pod="kuadrant-system/authorino-5bd69597d-7sb7p" Sep 11 14:12:52.573852 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:52.573854 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z9hc\" (UniqueName: \"kubernetes.io/projected/c8a29ab4-b886-48d2-9579-3e3a1dba2a78-kube-api-access-6z9hc\") pod \"authorino-5bd69597d-7sb7p\" (UID: \"c8a29ab4-b886-48d2-9579-3e3a1dba2a78\") " pod="kuadrant-system/authorino-5bd69597d-7sb7p" Sep 11 14:12:52.576479 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:52.576454 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8a29ab4-b886-48d2-9579-3e3a1dba2a78-tls-cert\") pod \"authorino-5bd69597d-7sb7p\" (UID: \"c8a29ab4-b886-48d2-9579-3e3a1dba2a78\") " pod="kuadrant-system/authorino-5bd69597d-7sb7p" Sep 11 14:12:52.597876 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:52.597841 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z9hc\" (UniqueName: \"kubernetes.io/projected/c8a29ab4-b886-48d2-9579-3e3a1dba2a78-kube-api-access-6z9hc\") pod \"authorino-5bd69597d-7sb7p\" (UID: \"c8a29ab4-b886-48d2-9579-3e3a1dba2a78\") " pod="kuadrant-system/authorino-5bd69597d-7sb7p" Sep 11 14:12:52.687298 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:52.687222 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bd69597d-7sb7p" Sep 11 14:12:52.966308 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:52.966278 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bd69597d-7sb7p"] Sep 11 14:12:52.966867 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:12:52.966846 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8a29ab4_b886_48d2_9579_3e3a1dba2a78.slice/crio-8ca0bc68cd490cf29138c12b6434e50be52d4ac13808a80fde75438203e0ee51 WatchSource:0}: Error finding container 8ca0bc68cd490cf29138c12b6434e50be52d4ac13808a80fde75438203e0ee51: Status 404 returned error can't find the container with id 8ca0bc68cd490cf29138c12b6434e50be52d4ac13808a80fde75438203e0ee51 Sep 11 14:12:53.679266 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:53.679220 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bd69597d-7sb7p" event={"ID":"c8a29ab4-b886-48d2-9579-3e3a1dba2a78","Type":"ContainerStarted","Data":"8ca0bc68cd490cf29138c12b6434e50be52d4ac13808a80fde75438203e0ee51"} Sep 11 14:12:54.689229 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:54.689182 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bd69597d-7sb7p" event={"ID":"c8a29ab4-b886-48d2-9579-3e3a1dba2a78","Type":"ContainerStarted","Data":"d6b45fac3c49473a999faa752d932e3038ae384f441205dad37deab6e6fbe1cd"} Sep 11 14:12:54.719632 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:54.719559 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5bd69597d-7sb7p" podStartSLOduration=1.660379495 podStartE2EDuration="2.719534369s" totalImagesPullingTime="1.059154874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:12:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:12:54.717541036 +0000 UTC m=+915.122522173" watchObservedRunningTime="2026-09-11 14:12:54.719534369 +0000 UTC m=+915.124515498" Sep 11 14:12:54.767360 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:54.767322 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-775884f7c9-zgk92"] Sep 11 14:12:54.767593 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:54.767558 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-775884f7c9-zgk92" podUID="84ffb829-1224-4aa6-af3a-660da4a9217a" containerName="authorino" containerID="cri-o://743d2b3d5942d96b093ebf51ea79ae5b30e5bb072adc9c902918bc5cfe734817" gracePeriod=30 Sep 11 14:12:55.046021 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.045991 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775884f7c9-zgk92" Sep 11 14:12:55.105409 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.105376 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84ffb829-1224-4aa6-af3a-660da4a9217a-tls-cert\") pod \"84ffb829-1224-4aa6-af3a-660da4a9217a\" (UID: \"84ffb829-1224-4aa6-af3a-660da4a9217a\") " Sep 11 14:12:55.105598 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.105429 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7jg6\" (UniqueName: \"kubernetes.io/projected/84ffb829-1224-4aa6-af3a-660da4a9217a-kube-api-access-p7jg6\") pod \"84ffb829-1224-4aa6-af3a-660da4a9217a\" (UID: \"84ffb829-1224-4aa6-af3a-660da4a9217a\") " Sep 11 14:12:55.107634 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.107606 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ffb829-1224-4aa6-af3a-660da4a9217a-kube-api-access-p7jg6" (OuterVolumeSpecName: "kube-api-access-p7jg6") pod "84ffb829-1224-4aa6-af3a-660da4a9217a" (UID: "84ffb829-1224-4aa6-af3a-660da4a9217a"). InnerVolumeSpecName "kube-api-access-p7jg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:12:55.118920 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.118887 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ffb829-1224-4aa6-af3a-660da4a9217a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "84ffb829-1224-4aa6-af3a-660da4a9217a" (UID: "84ffb829-1224-4aa6-af3a-660da4a9217a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:12:55.207016 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.206936 2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84ffb829-1224-4aa6-af3a-660da4a9217a-tls-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:12:55.207016 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.206975 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7jg6\" (UniqueName: \"kubernetes.io/projected/84ffb829-1224-4aa6-af3a-660da4a9217a-kube-api-access-p7jg6\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:12:55.696999 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.696966 2566 generic.go:356] "Generic (PLEG): container finished" podID="84ffb829-1224-4aa6-af3a-660da4a9217a" containerID="743d2b3d5942d96b093ebf51ea79ae5b30e5bb072adc9c902918bc5cfe734817" exitCode=0 Sep 11 14:12:55.697507 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.697008 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-775884f7c9-zgk92" Sep 11 14:12:55.697507 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.697055 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775884f7c9-zgk92" event={"ID":"84ffb829-1224-4aa6-af3a-660da4a9217a","Type":"ContainerDied","Data":"743d2b3d5942d96b093ebf51ea79ae5b30e5bb072adc9c902918bc5cfe734817"} Sep 11 14:12:55.697507 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.697124 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-775884f7c9-zgk92" event={"ID":"84ffb829-1224-4aa6-af3a-660da4a9217a","Type":"ContainerDied","Data":"3554ba69b8afcbb97962114d44a2cb443384bf7470927e911786975b57a432de"} Sep 11 14:12:55.697507 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.697146 2566 scope.go:117] "RemoveContainer" containerID="743d2b3d5942d96b093ebf51ea79ae5b30e5bb072adc9c902918bc5cfe734817" Sep 11 14:12:55.707274 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.707254 2566 scope.go:117] "RemoveContainer" containerID="743d2b3d5942d96b093ebf51ea79ae5b30e5bb072adc9c902918bc5cfe734817" Sep 11 14:12:55.707609 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:12:55.707587 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743d2b3d5942d96b093ebf51ea79ae5b30e5bb072adc9c902918bc5cfe734817\": container with ID starting with 743d2b3d5942d96b093ebf51ea79ae5b30e5bb072adc9c902918bc5cfe734817 not found: ID does not exist" containerID="743d2b3d5942d96b093ebf51ea79ae5b30e5bb072adc9c902918bc5cfe734817" Sep 11 14:12:55.707700 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.707619 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743d2b3d5942d96b093ebf51ea79ae5b30e5bb072adc9c902918bc5cfe734817"} err="failed to get container status \"743d2b3d5942d96b093ebf51ea79ae5b30e5bb072adc9c902918bc5cfe734817\": rpc error: code = NotFound desc = could not find container \"743d2b3d5942d96b093ebf51ea79ae5b30e5bb072adc9c902918bc5cfe734817\": container with ID starting with 743d2b3d5942d96b093ebf51ea79ae5b30e5bb072adc9c902918bc5cfe734817 not found: ID does not exist" Sep 11 14:12:55.725777 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.725746 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-775884f7c9-zgk92"] Sep 11 14:12:55.731968 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:55.731939 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-775884f7c9-zgk92"] Sep 11 14:12:56.098647 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:12:56.098616 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84ffb829-1224-4aa6-af3a-660da4a9217a" path="/var/lib/kubelet/pods/84ffb829-1224-4aa6-af3a-660da4a9217a/volumes" Sep 11 14:13:13.311988 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.311956 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8"] Sep 11 14:13:13.312479 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.312399 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84ffb829-1224-4aa6-af3a-660da4a9217a" containerName="authorino" Sep 11 14:13:13.312479 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.312415 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="84ffb829-1224-4aa6-af3a-660da4a9217a" containerName="authorino" Sep 11 14:13:13.312583 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.312498 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="84ffb829-1224-4aa6-af3a-660da4a9217a" containerName="authorino" Sep 11 14:13:13.315665 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.315642 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.318976 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.318955 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-be0c80-openshift-default-dockercfg-jmx8s\"" Sep 11 14:13:13.338734 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.338706 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8"] Sep 11 14:13:13.373923 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.373886 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istiod-ca-cert\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.373923 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.373926 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-podinfo\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.374155 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.373943 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xwjg\" (UniqueName: \"kubernetes.io/projected/fffb9f82-116a-4c91-bb26-3fd11261e4b8-kube-api-access-5xwjg\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.374155 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.374040 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-data\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.374155 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.374125 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-envoy\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.374155 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.374142 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-token\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.374416 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.374168 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-workload-socket\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.374416 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.374224 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-workload-certs\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.374416 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.374281 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-credential-socket\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.475579 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.475538 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-data\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.475771 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.475612 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-envoy\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.475771 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.475636 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-token\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.475771 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.475670 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-workload-socket\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.475771 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.475698 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-workload-certs\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.475999 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.475842 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-credential-socket\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.475999 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.475907 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istiod-ca-cert\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.475999 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.475950 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-podinfo\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.475999 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.475975 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xwjg\" (UniqueName: \"kubernetes.io/projected/fffb9f82-116a-4c91-bb26-3fd11261e4b8-kube-api-access-5xwjg\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.475999 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.475980 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-data\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.476339 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.476151 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-workload-certs\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.476339 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.476271 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-credential-socket\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.476339 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.476320 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-workload-socket\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.476792 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.476764 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istiod-ca-cert\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.478261 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.478240 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-envoy\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.479001 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.478951 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-podinfo\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.495759 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.495732 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-token\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.495984 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.495967 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xwjg\" (UniqueName: \"kubernetes.io/projected/fffb9f82-116a-4c91-bb26-3fd11261e4b8-kube-api-access-5xwjg\") pod \"e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.628732 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.628648 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:13.914943 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:13:13.914907 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffb9f82_116a_4c91_bb26_3fd11261e4b8.slice/crio-2ca6445a0db2a3bfcb9a3218b48ab6765870ade3911394f274a66ef5240ae79e WatchSource:0}: Error finding container 2ca6445a0db2a3bfcb9a3218b48ab6765870ade3911394f274a66ef5240ae79e: Status 404 returned error can't find the container with id 2ca6445a0db2a3bfcb9a3218b48ab6765870ade3911394f274a66ef5240ae79e Sep 11 14:13:13.915488 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.915464 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8"] Sep 11 14:13:13.917323 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.917289 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:13.917420 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.917357 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:13.917420 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:13.917382 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:14.814843 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:14.814806 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" event={"ID":"fffb9f82-116a-4c91-bb26-3fd11261e4b8","Type":"ContainerStarted","Data":"84b1bfd6177963b651d10d9b71b716056acad4d6eaa8739e8cfcc5fbb89cda23"} Sep 11 14:13:14.815273 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:14.814870 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" event={"ID":"fffb9f82-116a-4c91-bb26-3fd11261e4b8","Type":"ContainerStarted","Data":"2ca6445a0db2a3bfcb9a3218b48ab6765870ade3911394f274a66ef5240ae79e"} Sep 11 14:13:14.849985 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:14.849929 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" podStartSLOduration=1.849911896 podStartE2EDuration="1.849911896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:14.846520574 +0000 UTC m=+935.251501702" watchObservedRunningTime="2026-09-11 14:13:14.849911896 +0000 UTC m=+935.254893023" Sep 11 14:13:15.629807 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:15.629771 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:15.634781 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:15.634758 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:15.821240 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:15.821209 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:15.822209 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:15.822189 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:22.016488 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.016447 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz"] Sep 11 14:13:22.022095 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.022055 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.029046 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.029011 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-2a21d9-openshift-default-dockercfg-n2xp7\"" Sep 11 14:13:22.048570 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.048543 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-envoy\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.048731 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.048588 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvsv\" (UniqueName: \"kubernetes.io/projected/ad25e1d0-b107-4042-86d0-9026e5c78517-kube-api-access-nvvsv\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.048731 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.048606 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad25e1d0-b107-4042-86d0-9026e5c78517-istiod-ca-cert\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.048731 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.048707 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-token\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.048869 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.048745 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-credential-socket\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.048869 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.048778 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-data\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.048869 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.048859 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-workload-certs\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.048992 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.048897 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-workload-socket\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.048992 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.048914 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-podinfo\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.069819 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.069780 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz"] Sep 11 14:13:22.149604 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.149562 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad25e1d0-b107-4042-86d0-9026e5c78517-istiod-ca-cert\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.149804 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.149616 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-token\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.149804 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.149643 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-credential-socket\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.149804 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.149668 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-data\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.149804 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.149714 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-workload-certs\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.149804 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.149748 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-workload-socket\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.149804 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.149777 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-podinfo\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.149804 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.149804 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-envoy\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.150224 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.149863 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvsv\" (UniqueName: \"kubernetes.io/projected/ad25e1d0-b107-4042-86d0-9026e5c78517-kube-api-access-nvvsv\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.150224 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.150118 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-credential-socket\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.150224 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.150188 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-data\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.150344 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.150226 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-workload-socket\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.150344 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.150260 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-workload-certs\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.150442 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.150418 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad25e1d0-b107-4042-86d0-9026e5c78517-istiod-ca-cert\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.152768 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.152746 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-podinfo\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.152768 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.152762 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-envoy\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.174444 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.174420 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-token\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.236213 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.236177 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvsv\" (UniqueName: \"kubernetes.io/projected/ad25e1d0-b107-4042-86d0-9026e5c78517-kube-api-access-nvvsv\") pod \"e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.333570 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.333479 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:22.634329 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.634131 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz"] Sep 11 14:13:22.639948 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.639910 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:22.640055 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.640001 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:22.640055 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.640041 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:22.652693 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.652667 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt"] Sep 11 14:13:22.657430 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.657412 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.660573 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.660551 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1dc719b7-openshift-default-dockercfg-9lqw6\"" Sep 11 14:13:22.670776 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.670752 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt"] Sep 11 14:13:22.755839 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.755792 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-token\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.755999 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.755847 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6qft\" (UniqueName: \"kubernetes.io/projected/759456e0-a5a2-4d54-8938-a9e6730693f5-kube-api-access-d6qft\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.755999 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.755915 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-workload-certs\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.755999 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.755945 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-data\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.756212 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.756076 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-workload-socket\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.756212 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.756198 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759456e0-a5a2-4d54-8938-a9e6730693f5-istiod-ca-cert\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.756325 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.756296 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-credential-socket\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.756434 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.756397 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-podinfo\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.756542 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.756457 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-envoy\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.857003 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.856969 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-envoy\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.857206 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.857036 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-token\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.857206 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.857055 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6qft\" (UniqueName: \"kubernetes.io/projected/759456e0-a5a2-4d54-8938-a9e6730693f5-kube-api-access-d6qft\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.857206 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.857077 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-workload-certs\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.857206 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.857109 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-data\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.857206 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.857126 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-workload-socket\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.857206 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.857141 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759456e0-a5a2-4d54-8938-a9e6730693f5-istiod-ca-cert\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.857206 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.857184 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-credential-socket\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.857206 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.857205 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-podinfo\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.858237 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.857667 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-data\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.858237 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.857982 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-workload-socket\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.858237 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.858057 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-workload-certs\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.858237 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.858205 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-credential-socket\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.858551 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.858364 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759456e0-a5a2-4d54-8938-a9e6730693f5-istiod-ca-cert\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.860147 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.860123 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-podinfo\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.860754 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.860728 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-envoy\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.862789 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.862760 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" event={"ID":"ad25e1d0-b107-4042-86d0-9026e5c78517","Type":"ContainerStarted","Data":"8eda68b74570ac1e091bac83252dfc2e584ce66c1fab5746cc80cecddca8bba6"} Sep 11 14:13:22.862922 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.862815 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" event={"ID":"ad25e1d0-b107-4042-86d0-9026e5c78517","Type":"ContainerStarted","Data":"7ad7386056bf73f91346f4a7ce72d86a781ebb3dd096a1ecb799e5db30fbd7d0"} Sep 11 14:13:22.866918 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.866895 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-token\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.869180 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.869153 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6qft\" (UniqueName: \"kubernetes.io/projected/759456e0-a5a2-4d54-8938-a9e6730693f5-kube-api-access-d6qft\") pod \"e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:22.896035 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.895944 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" podStartSLOduration=1.89592369 podStartE2EDuration="1.89592369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:22.894220069 +0000 UTC m=+943.299201198" watchObservedRunningTime="2026-09-11 14:13:22.89592369 +0000 UTC m=+943.300904820" Sep 11 14:13:22.980634 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:22.980600 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:23.275143 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:23.275112 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt"] Sep 11 14:13:23.276044 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:13:23.276013 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759456e0_a5a2_4d54_8938_a9e6730693f5.slice/crio-9215cd5637c491e45a3f1b399e7ea2ea1260e70d226b293c576cc526850013c6 WatchSource:0}: Error finding container 9215cd5637c491e45a3f1b399e7ea2ea1260e70d226b293c576cc526850013c6: Status 404 returned error can't find the container with id 9215cd5637c491e45a3f1b399e7ea2ea1260e70d226b293c576cc526850013c6 Sep 11 14:13:23.278406 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:23.278373 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:23.278485 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:23.278438 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:23.278485 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:23.278469 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:23.333847 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:23.333828 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:23.339441 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:23.339415 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:23.871662 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:23.871624 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" event={"ID":"759456e0-a5a2-4d54-8938-a9e6730693f5","Type":"ContainerStarted","Data":"33a8a23cfdfa601d15fc112cfc7cc80b82eaf306045b573ba8bba3d06bb1b08c"} Sep 11 14:13:23.871833 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:23.871690 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" event={"ID":"759456e0-a5a2-4d54-8938-a9e6730693f5","Type":"ContainerStarted","Data":"9215cd5637c491e45a3f1b399e7ea2ea1260e70d226b293c576cc526850013c6"} Sep 11 14:13:23.871833 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:23.871812 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:23.872902 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:23.872872 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:23.905096 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:23.905019 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" podStartSLOduration=1.904995759 podStartE2EDuration="1.904995759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:23.899400075 +0000 UTC m=+944.304381205" watchObservedRunningTime="2026-09-11 14:13:23.904995759 +0000 UTC m=+944.309976889" Sep 11 14:13:23.980834 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:23.980809 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:24.986541 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:24.986512 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:25.883842 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:25.883809 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:25.884926 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:25.884901 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:31.864253 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:31.864213 2566 patch_prober.go:28] interesting pod/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 14:13:31.864678 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:31.864285 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" podUID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 14:13:32.295295 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.295263 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55"] Sep 11 14:13:32.307027 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.306994 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.310168 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.310145 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c6156e-openshift-default-dockercfg-vdh8w\"" Sep 11 14:13:32.319851 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.319825 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55"] Sep 11 14:13:32.374378 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.374345 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-envoy\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.374378 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.374379 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmpwm\" (UniqueName: \"kubernetes.io/projected/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-kube-api-access-zmpwm\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.374587 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.374403 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-workload-socket\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.374587 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.374500 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-data\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.374587 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.374550 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istiod-ca-cert\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.374587 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.374572 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-token\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.374713 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.374608 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-podinfo\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.374713 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.374648 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-credential-socket\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.374713 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.374667 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-workload-certs\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.475881 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.475843 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmpwm\" (UniqueName: \"kubernetes.io/projected/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-kube-api-access-zmpwm\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.475881 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.475885 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-workload-socket\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.476169 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.475923 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-data\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.476169 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.475953 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istiod-ca-cert\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.476169 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.475994 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-token\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.476169 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.476022 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-podinfo\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.476169 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.476047 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-credential-socket\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.476169 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.476075 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-workload-certs\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.476473 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.476172 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-envoy\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.476473 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.476433 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-workload-socket\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.476580 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.476511 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-data\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.476640 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.476595 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-credential-socket\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.476640 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.476617 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-workload-certs\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.477034 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.477014 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istiod-ca-cert\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.478523 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.478494 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-envoy\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.478723 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.478704 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-podinfo\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.489499 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.489471 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmpwm\" (UniqueName: \"kubernetes.io/projected/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-kube-api-access-zmpwm\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.491756 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.491733 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-token\") pod \"e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.619374 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.619292 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:32.911618 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.911588 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55"] Sep 11 14:13:32.913655 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:13:32.913366 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0878d1_48b2_4cd2_865b_c6f21e95a0a3.slice/crio-15f2593f5a27c30cea5f284efff74c7e01b2fdbbb8e922caf8e6b6bc6af80ac6 WatchSource:0}: Error finding container 15f2593f5a27c30cea5f284efff74c7e01b2fdbbb8e922caf8e6b6bc6af80ac6: Status 404 returned error can't find the container with id 15f2593f5a27c30cea5f284efff74c7e01b2fdbbb8e922caf8e6b6bc6af80ac6 Sep 11 14:13:32.916774 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.916712 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:32.916907 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.916814 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:32.916907 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.916863 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:32.969346 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:32.969129 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" event={"ID":"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3","Type":"ContainerStarted","Data":"15f2593f5a27c30cea5f284efff74c7e01b2fdbbb8e922caf8e6b6bc6af80ac6"} Sep 11 14:13:33.977033 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:33.977000 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" event={"ID":"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3","Type":"ContainerStarted","Data":"d20182eacb3efcc536b185de20c41ef5cc9d8d7062af208380b975e04ef9c8c6"} Sep 11 14:13:33.978740 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:33.978718 2566 generic.go:356] "Generic (PLEG): container finished" podID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" containerID="84b1bfd6177963b651d10d9b71b716056acad4d6eaa8739e8cfcc5fbb89cda23" exitCode=0 Sep 11 14:13:33.978843 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:33.978778 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" event={"ID":"fffb9f82-116a-4c91-bb26-3fd11261e4b8","Type":"ContainerDied","Data":"84b1bfd6177963b651d10d9b71b716056acad4d6eaa8739e8cfcc5fbb89cda23"} Sep 11 14:13:33.979133 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:33.979121 2566 scope.go:117] "RemoveContainer" containerID="84b1bfd6177963b651d10d9b71b716056acad4d6eaa8739e8cfcc5fbb89cda23" Sep 11 14:13:33.980157 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:33.980124 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:33.980269 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:33.980195 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:33.980269 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:33.980226 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:34.043605 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:34.043550 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" podStartSLOduration=2.043527955 podStartE2EDuration="2.043527955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:34.04031047 +0000 UTC m=+954.445291610" watchObservedRunningTime="2026-09-11 14:13:34.043527955 +0000 UTC m=+954.448509083" Sep 11 14:13:34.621987 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:34.621952 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:34.638259 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:34.638221 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:34.988738 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:34.988701 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" event={"ID":"fffb9f82-116a-4c91-bb26-3fd11261e4b8","Type":"ContainerStarted","Data":"17cda0f3900842eef6ccf895cb567409d84f563c1f925bc51c43c2fb0f4edccc"} Sep 11 14:13:34.989269 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:34.989001 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:34.990145 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:34.990128 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:13:35.630112 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:35.629224 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:35.630112 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:35.629378 2566 patch_prober.go:28] interesting pod/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 11 14:13:35.630112 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:35.629449 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" podUID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 11 14:13:36.629387 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:36.629334 2566 patch_prober.go:28] interesting pod/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 11 14:13:36.629919 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:36.629413 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" podUID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 11 14:13:37.630207 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:37.630165 2566 patch_prober.go:28] interesting pod/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 11 14:13:37.630792 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:37.630754 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" podUID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 11 14:13:38.629370 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:38.629330 2566 patch_prober.go:28] interesting pod/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 11 14:13:38.629689 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:38.629652 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" podUID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 11 14:13:39.633111 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:39.629262 2566 patch_prober.go:28] interesting pod/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 11 14:13:39.633111 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:39.629357 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" podUID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 11 14:13:40.633111 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:40.633053 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:40.633400 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:40.633373 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:40.634280 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:40.634261 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:13:41.925286 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:41.925233 2566 patch_prober.go:28] interesting pod/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 14:13:41.925728 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:41.925299 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" podUID="759456e0-a5a2-4d54-8938-a9e6730693f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 14:13:44.063555 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:44.063527 2566 generic.go:356] "Generic (PLEG): container finished" podID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerID="8eda68b74570ac1e091bac83252dfc2e584ce66c1fab5746cc80cecddca8bba6" exitCode=0 Sep 11 14:13:44.063976 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:44.063594 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" event={"ID":"ad25e1d0-b107-4042-86d0-9026e5c78517","Type":"ContainerDied","Data":"8eda68b74570ac1e091bac83252dfc2e584ce66c1fab5746cc80cecddca8bba6"} Sep 11 14:13:44.064044 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:44.064029 2566 scope.go:117] "RemoveContainer" containerID="8eda68b74570ac1e091bac83252dfc2e584ce66c1fab5746cc80cecddca8bba6" Sep 11 14:13:44.065295 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:44.065264 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:44.065391 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:44.065337 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:44.065391 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:44.065375 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:44.065532 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:44.065515 2566 generic.go:356] "Generic (PLEG): container finished" podID="759456e0-a5a2-4d54-8938-a9e6730693f5" containerID="33a8a23cfdfa601d15fc112cfc7cc80b82eaf306045b573ba8bba3d06bb1b08c" exitCode=0 Sep 11 14:13:44.065584 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:44.065569 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" event={"ID":"759456e0-a5a2-4d54-8938-a9e6730693f5","Type":"ContainerDied","Data":"33a8a23cfdfa601d15fc112cfc7cc80b82eaf306045b573ba8bba3d06bb1b08c"} Sep 11 14:13:44.065896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:44.065884 2566 scope.go:117] "RemoveContainer" containerID="33a8a23cfdfa601d15fc112cfc7cc80b82eaf306045b573ba8bba3d06bb1b08c" Sep 11 14:13:44.066785 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:44.066762 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:44.066851 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:44.066821 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:44.066895 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:44.066852 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:45.076051 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:45.076016 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" event={"ID":"759456e0-a5a2-4d54-8938-a9e6730693f5","Type":"ContainerStarted","Data":"2e28bea09cdcabfd63ae157f141bfa2619a1a704e07a0922a5945bace603c21b"} Sep 11 14:13:45.077994 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:45.077970 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" event={"ID":"ad25e1d0-b107-4042-86d0-9026e5c78517","Type":"ContainerStarted","Data":"aef628c492f57a195686e3b7713a21b5b011fbfaebeef7dfbc24744fb8678d47"} Sep 11 14:13:45.334568 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:45.334470 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:45.334837 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:45.334700 2566 patch_prober.go:28] interesting pod/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 11 14:13:45.335052 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:45.334988 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" podUID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 11 14:13:45.911107 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:45.911041 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm"] Sep 11 14:13:45.915728 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:45.915702 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm" Sep 11 14:13:45.919736 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:45.919710 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:13:45.929326 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:45.929290 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm"] Sep 11 14:13:45.981507 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:45.981470 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:45.981974 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:45.981950 2566 patch_prober.go:28] interesting pod/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 11 14:13:45.982159 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:45.982139 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" podUID="759456e0-a5a2-4d54-8938-a9e6730693f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 11 14:13:46.043165 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:46.042642 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm\" (UID: \"e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm" Sep 11 14:13:46.143730 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:46.143676 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm\" (UID: \"e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm" Sep 11 14:13:46.144616 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:46.144586 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm\" (UID: \"e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm" Sep 11 14:13:46.237761 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:46.237718 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm" Sep 11 14:13:46.358634 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:46.341464 2566 patch_prober.go:28] interesting pod/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 11 14:13:46.358634 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:46.341527 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" podUID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 11 14:13:46.538238 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:46.538143 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm"] Sep 11 14:13:46.553795 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:46.553169 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:13:46.981115 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:46.981051 2566 patch_prober.go:28] interesting pod/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 11 14:13:46.981308 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:46.981131 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" podUID="759456e0-a5a2-4d54-8938-a9e6730693f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 11 14:13:47.092151 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:47.092112 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm" event={"ID":"e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e","Type":"ContainerStarted","Data":"e556d91c9d776f55b2f4b96fd8c8bb08e4f0ed1b9ed39fa2b735ee60358f4095"} Sep 11 14:13:47.334613 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:47.334514 2566 patch_prober.go:28] interesting pod/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 11 14:13:47.335620 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:47.335197 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" podUID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 11 14:13:47.981550 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:47.981505 2566 patch_prober.go:28] interesting pod/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 11 14:13:47.981731 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:47.981597 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" podUID="759456e0-a5a2-4d54-8938-a9e6730693f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 11 14:13:48.348166 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:48.347212 2566 patch_prober.go:28] interesting pod/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 11 14:13:48.348166 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:48.347278 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" podUID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 11 14:13:48.981364 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:48.981320 2566 patch_prober.go:28] interesting pod/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 11 14:13:48.981590 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:48.981391 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" podUID="759456e0-a5a2-4d54-8938-a9e6730693f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 11 14:13:49.337061 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:49.336528 2566 patch_prober.go:28] interesting pod/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 11 14:13:49.337061 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:49.336595 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" podUID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 11 14:13:49.981071 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:49.980980 2566 patch_prober.go:28] interesting pod/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 11 14:13:49.981071 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:49.981050 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" podUID="759456e0-a5a2-4d54-8938-a9e6730693f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 11 14:13:50.334169 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:50.333935 2566 patch_prober.go:28] interesting pod/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Sep 11 14:13:50.334169 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:50.334005 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" podUID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Sep 11 14:13:50.987532 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:50.987492 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:50.988049 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:50.988029 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:50.989012 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:50.988990 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:13:51.156169 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:51.156130 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm" event={"ID":"e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e","Type":"ContainerStarted","Data":"d9cd362d4d85b62f5e2eea00f2bbab4c5c68c14086881d0f0b0c87ccc220a718"} Sep 11 14:13:51.180778 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:51.180730 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm" podStartSLOduration=2.064088582 podStartE2EDuration="6.180713133s" totalImagesPullingTime="4.116624551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:51.177008689 +0000 UTC m=+971.581989818" watchObservedRunningTime="2026-09-11 14:13:51.180713133 +0000 UTC m=+971.585694261" Sep 11 14:13:51.338917 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:51.338829 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:51.339579 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:51.339532 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:51.340694 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:51.340667 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:13:52.163499 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:52.163460 2566 generic.go:356] "Generic (PLEG): container finished" podID="e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e" containerID="d9cd362d4d85b62f5e2eea00f2bbab4c5c68c14086881d0f0b0c87ccc220a718" exitCode=0 Sep 11 14:13:52.163902 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:52.163550 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm" event={"ID":"e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e","Type":"ContainerDied","Data":"d9cd362d4d85b62f5e2eea00f2bbab4c5c68c14086881d0f0b0c87ccc220a718"} Sep 11 14:13:53.360514 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:53.360485 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm" Sep 11 14:13:53.447672 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:53.447583 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e-maas-api-service-ca\") pod \"e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e\" (UID: \"e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e\") " Sep 11 14:13:53.447955 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:53.447928 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e" (UID: "e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:13:53.548952 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:53.548917 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:13:54.177905 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:54.177867 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm" Sep 11 14:13:54.177905 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:54.177890 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm" event={"ID":"e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e","Type":"ContainerDied","Data":"e556d91c9d776f55b2f4b96fd8c8bb08e4f0ed1b9ed39fa2b735ee60358f4095"} Sep 11 14:13:54.178121 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:54.177941 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e556d91c9d776f55b2f4b96fd8c8bb08e4f0ed1b9ed39fa2b735ee60358f4095" Sep 11 14:13:58.061309 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:58.061278 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm"] Sep 11 14:13:58.065786 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:58.065758 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1dc719b7-tx4nm"] Sep 11 14:13:58.113279 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:58.113240 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e" path="/var/lib/kubelet/pods/e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e/volumes" Sep 11 14:13:58.216869 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:58.216835 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt"] Sep 11 14:13:58.217185 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:58.217153 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" podUID="759456e0-a5a2-4d54-8938-a9e6730693f5" containerName="istio-proxy" containerID="cri-o://2e28bea09cdcabfd63ae157f141bfa2619a1a704e07a0922a5945bace603c21b" gracePeriod=30 Sep 11 14:13:59.712342 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.712307 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc"] Sep 11 14:13:59.712872 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.712786 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e" containerName="revoke-keys" Sep 11 14:13:59.712872 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.712805 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e" containerName="revoke-keys" Sep 11 14:13:59.712980 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.712932 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e597ecdd-56fe-4f02-ba5d-d0bc0114ab0e" containerName="revoke-keys" Sep 11 14:13:59.728644 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.728612 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.734962 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.734932 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-59484b3d-gw-openshift-default-dockercfg-42r8b\"" Sep 11 14:13:59.746695 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.746666 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc"] Sep 11 14:13:59.812988 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.812945 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-token\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.813231 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.813004 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-data\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.813231 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.813119 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-credential-socket\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.813231 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.813170 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-envoy\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.813231 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.813212 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-workload-certs\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.813472 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.813251 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-podinfo\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.813472 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.813275 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-workload-socket\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.813472 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.813326 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpnqt\" (UniqueName: \"kubernetes.io/projected/7fd727e7-9b16-4ba3-aea7-771a1db8e913-kube-api-access-wpnqt\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.813472 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.813369 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istiod-ca-cert\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.914303 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.914270 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-workload-certs\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.914303 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.914307 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-podinfo\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.914533 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.914329 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-workload-socket\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.914533 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.914364 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpnqt\" (UniqueName: \"kubernetes.io/projected/7fd727e7-9b16-4ba3-aea7-771a1db8e913-kube-api-access-wpnqt\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.914533 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.914402 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istiod-ca-cert\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.914533 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.914442 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-token\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.914533 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.914468 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-data\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.914793 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.914552 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-credential-socket\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.914793 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.914594 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-envoy\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.914793 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.914740 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-workload-socket\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.914793 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.914758 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-workload-certs\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.915036 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.915006 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-data\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.915209 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.915187 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istiod-ca-cert\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.915285 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.915264 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-credential-socket\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.916818 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.916783 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-envoy\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.917039 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.917025 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-podinfo\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.937401 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.937369 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-token\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:13:59.940166 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:13:59.940133 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpnqt\" (UniqueName: \"kubernetes.io/projected/7fd727e7-9b16-4ba3-aea7-771a1db8e913-kube-api-access-wpnqt\") pod \"e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:14:00.040212 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:00.040129 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:14:00.217290 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:00.217252 2566 generic.go:356] "Generic (PLEG): container finished" podID="5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" containerID="d20182eacb3efcc536b185de20c41ef5cc9d8d7062af208380b975e04ef9c8c6" exitCode=0 Sep 11 14:14:00.217453 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:00.217320 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" event={"ID":"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3","Type":"ContainerDied","Data":"d20182eacb3efcc536b185de20c41ef5cc9d8d7062af208380b975e04ef9c8c6"} Sep 11 14:14:00.217702 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:00.217689 2566 scope.go:117] "RemoveContainer" containerID="d20182eacb3efcc536b185de20c41ef5cc9d8d7062af208380b975e04ef9c8c6" Sep 11 14:14:00.218777 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:00.218742 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:14:00.218893 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:00.218818 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:14:00.218893 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:00.218856 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:14:00.351241 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:00.351159 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc"] Sep 11 14:14:00.358542 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:14:00.358505 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd727e7_9b16_4ba3_aea7_771a1db8e913.slice/crio-d60c1980aeacd0b90c043a52ee61ce53ceb0f86c6023a511123a25770b29e38d WatchSource:0}: Error finding container d60c1980aeacd0b90c043a52ee61ce53ceb0f86c6023a511123a25770b29e38d: Status 404 returned error can't find the container with id d60c1980aeacd0b90c043a52ee61ce53ceb0f86c6023a511123a25770b29e38d Sep 11 14:14:00.361326 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:00.361294 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:14:00.361451 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:00.361375 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:14:00.361451 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:00.361417 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:14:01.230480 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:01.230431 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" event={"ID":"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3","Type":"ContainerStarted","Data":"f8373d228c54125a4ed5d1dfa09f501dbeece8f3f5e3840b0bc487bba2f3aebf"} Sep 11 14:14:01.233553 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:01.233523 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" event={"ID":"7fd727e7-9b16-4ba3-aea7-771a1db8e913","Type":"ContainerStarted","Data":"efb87c517c021a045e24fafbc144a5e8f03f5f15f9fd4c99a28fb7ff67365a79"} Sep 11 14:14:01.233680 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:01.233570 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" event={"ID":"7fd727e7-9b16-4ba3-aea7-771a1db8e913","Type":"ContainerStarted","Data":"d60c1980aeacd0b90c043a52ee61ce53ceb0f86c6023a511123a25770b29e38d"} Sep 11 14:14:01.619924 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:01.619524 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:14:01.621539 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:01.621190 2566 patch_prober.go:28] interesting pod/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 11 14:14:01.621539 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:01.621251 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" podUID="5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 11 14:14:01.691200 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:01.691145 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" podStartSLOduration=2.691126047 podStartE2EDuration="2.691126047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:14:01.684285477 +0000 UTC m=+982.089266604" watchObservedRunningTime="2026-09-11 14:14:01.691126047 +0000 UTC m=+982.096107174" Sep 11 14:14:02.040394 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:02.040354 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:14:02.045397 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:02.045373 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:14:02.240058 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:02.240026 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:14:02.241147 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:02.241130 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:14:02.620278 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:02.620235 2566 patch_prober.go:28] interesting pod/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 11 14:14:02.620503 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:02.620307 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" podUID="5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 11 14:14:03.533635 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.533610 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:14:03.622108 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.621540 2566 patch_prober.go:28] interesting pod/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 11 14:14:03.622108 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.621607 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" podUID="5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 11 14:14:03.655502 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.655469 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-workload-socket\") pod \"759456e0-a5a2-4d54-8938-a9e6730693f5\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " Sep 11 14:14:03.655669 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.655511 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6qft\" (UniqueName: \"kubernetes.io/projected/759456e0-a5a2-4d54-8938-a9e6730693f5-kube-api-access-d6qft\") pod \"759456e0-a5a2-4d54-8938-a9e6730693f5\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " Sep 11 14:14:03.655669 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.655534 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-data\") pod \"759456e0-a5a2-4d54-8938-a9e6730693f5\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " Sep 11 14:14:03.655669 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.655564 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759456e0-a5a2-4d54-8938-a9e6730693f5-istiod-ca-cert\") pod \"759456e0-a5a2-4d54-8938-a9e6730693f5\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " Sep 11 14:14:03.655669 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.655592 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-credential-socket\") pod \"759456e0-a5a2-4d54-8938-a9e6730693f5\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " Sep 11 14:14:03.655669 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.655622 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-podinfo\") pod \"759456e0-a5a2-4d54-8938-a9e6730693f5\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " Sep 11 14:14:03.655854 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.655673 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-workload-certs\") pod \"759456e0-a5a2-4d54-8938-a9e6730693f5\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " Sep 11 14:14:03.655854 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.655714 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "759456e0-a5a2-4d54-8938-a9e6730693f5" (UID: "759456e0-a5a2-4d54-8938-a9e6730693f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:14:03.655854 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.655733 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-envoy\") pod \"759456e0-a5a2-4d54-8938-a9e6730693f5\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " Sep 11 14:14:03.655854 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.655772 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-token\") pod \"759456e0-a5a2-4d54-8938-a9e6730693f5\" (UID: \"759456e0-a5a2-4d54-8938-a9e6730693f5\") " Sep 11 14:14:03.655854 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.655826 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "759456e0-a5a2-4d54-8938-a9e6730693f5" (UID: "759456e0-a5a2-4d54-8938-a9e6730693f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:14:03.656161 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.655926 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759456e0-a5a2-4d54-8938-a9e6730693f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "759456e0-a5a2-4d54-8938-a9e6730693f5" (UID: "759456e0-a5a2-4d54-8938-a9e6730693f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:14:03.656161 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.655989 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "759456e0-a5a2-4d54-8938-a9e6730693f5" (UID: "759456e0-a5a2-4d54-8938-a9e6730693f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:14:03.656161 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.656063 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "759456e0-a5a2-4d54-8938-a9e6730693f5" (UID: "759456e0-a5a2-4d54-8938-a9e6730693f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:14:03.656551 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.656527 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759456e0-a5a2-4d54-8938-a9e6730693f5-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:03.656551 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.656548 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:03.656854 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.656557 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:03.656854 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.656566 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:03.656854 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.656573 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:03.658146 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.658122 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "759456e0-a5a2-4d54-8938-a9e6730693f5" (UID: "759456e0-a5a2-4d54-8938-a9e6730693f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:14:03.658383 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.658360 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "759456e0-a5a2-4d54-8938-a9e6730693f5" (UID: "759456e0-a5a2-4d54-8938-a9e6730693f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:14:03.658514 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.658495 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "759456e0-a5a2-4d54-8938-a9e6730693f5" (UID: "759456e0-a5a2-4d54-8938-a9e6730693f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:14:03.658558 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.658498 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759456e0-a5a2-4d54-8938-a9e6730693f5-kube-api-access-d6qft" (OuterVolumeSpecName: "kube-api-access-d6qft") pod "759456e0-a5a2-4d54-8938-a9e6730693f5" (UID: "759456e0-a5a2-4d54-8938-a9e6730693f5"). InnerVolumeSpecName "kube-api-access-d6qft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:14:03.757920 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.757839 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6qft\" (UniqueName: \"kubernetes.io/projected/759456e0-a5a2-4d54-8938-a9e6730693f5-kube-api-access-d6qft\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:03.757920 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.757869 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:03.757920 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.757878 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:03.757920 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:03.757886 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759456e0-a5a2-4d54-8938-a9e6730693f5-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:04.254902 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:04.254871 2566 generic.go:356] "Generic (PLEG): container finished" podID="759456e0-a5a2-4d54-8938-a9e6730693f5" containerID="2e28bea09cdcabfd63ae157f141bfa2619a1a704e07a0922a5945bace603c21b" exitCode=0 Sep 11 14:14:04.255100 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:04.254920 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" Sep 11 14:14:04.255100 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:04.254954 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" event={"ID":"759456e0-a5a2-4d54-8938-a9e6730693f5","Type":"ContainerDied","Data":"2e28bea09cdcabfd63ae157f141bfa2619a1a704e07a0922a5945bace603c21b"} Sep 11 14:14:04.255100 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:04.255002 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt" event={"ID":"759456e0-a5a2-4d54-8938-a9e6730693f5","Type":"ContainerDied","Data":"9215cd5637c491e45a3f1b399e7ea2ea1260e70d226b293c576cc526850013c6"} Sep 11 14:14:04.255100 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:04.255025 2566 scope.go:117] "RemoveContainer" containerID="2e28bea09cdcabfd63ae157f141bfa2619a1a704e07a0922a5945bace603c21b" Sep 11 14:14:04.264554 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:04.264526 2566 scope.go:117] "RemoveContainer" containerID="33a8a23cfdfa601d15fc112cfc7cc80b82eaf306045b573ba8bba3d06bb1b08c" Sep 11 14:14:04.273338 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:04.273321 2566 scope.go:117] "RemoveContainer" containerID="2e28bea09cdcabfd63ae157f141bfa2619a1a704e07a0922a5945bace603c21b" Sep 11 14:14:04.273588 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:14:04.273572 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e28bea09cdcabfd63ae157f141bfa2619a1a704e07a0922a5945bace603c21b\": container with ID starting with 2e28bea09cdcabfd63ae157f141bfa2619a1a704e07a0922a5945bace603c21b not found: ID does not exist" containerID="2e28bea09cdcabfd63ae157f141bfa2619a1a704e07a0922a5945bace603c21b" Sep 11 14:14:04.273684 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:04.273666 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e28bea09cdcabfd63ae157f141bfa2619a1a704e07a0922a5945bace603c21b"} err="failed to get container status \"2e28bea09cdcabfd63ae157f141bfa2619a1a704e07a0922a5945bace603c21b\": rpc error: code = NotFound desc = could not find container \"2e28bea09cdcabfd63ae157f141bfa2619a1a704e07a0922a5945bace603c21b\": container with ID starting with 2e28bea09cdcabfd63ae157f141bfa2619a1a704e07a0922a5945bace603c21b not found: ID does not exist" Sep 11 14:14:04.273751 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:04.273743 2566 scope.go:117] "RemoveContainer" containerID="33a8a23cfdfa601d15fc112cfc7cc80b82eaf306045b573ba8bba3d06bb1b08c" Sep 11 14:14:04.274037 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:14:04.274021 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a8a23cfdfa601d15fc112cfc7cc80b82eaf306045b573ba8bba3d06bb1b08c\": container with ID starting with 33a8a23cfdfa601d15fc112cfc7cc80b82eaf306045b573ba8bba3d06bb1b08c not found: ID does not exist" containerID="33a8a23cfdfa601d15fc112cfc7cc80b82eaf306045b573ba8bba3d06bb1b08c" Sep 11 14:14:04.274133 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:04.274045 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a8a23cfdfa601d15fc112cfc7cc80b82eaf306045b573ba8bba3d06bb1b08c"} err="failed to get container status \"33a8a23cfdfa601d15fc112cfc7cc80b82eaf306045b573ba8bba3d06bb1b08c\": rpc error: code = NotFound desc = could not find container \"33a8a23cfdfa601d15fc112cfc7cc80b82eaf306045b573ba8bba3d06bb1b08c\": container with ID starting with 33a8a23cfdfa601d15fc112cfc7cc80b82eaf306045b573ba8bba3d06bb1b08c not found: ID does not exist" Sep 11 14:14:04.291368 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:04.291331 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt"] Sep 11 14:14:04.294020 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:04.294001 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1dc719b7-openshift-default-54bb75d7fb-8tgmt"] Sep 11 14:14:04.620565 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:04.620524 2566 patch_prober.go:28] interesting pod/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 11 14:14:04.621111 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:04.620596 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" podUID="5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 11 14:14:04.989532 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:04.989504 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:14:05.620194 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:05.620145 2566 patch_prober.go:28] interesting pod/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Sep 11 14:14:05.620416 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:05.620218 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" podUID="5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Sep 11 14:14:06.099287 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:06.099250 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="759456e0-a5a2-4d54-8938-a9e6730693f5" path="/var/lib/kubelet/pods/759456e0-a5a2-4d54-8938-a9e6730693f5/volumes" Sep 11 14:14:06.663696 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:06.663667 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:14:06.665736 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:06.665710 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:14:23.379237 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:23.379202 2566 generic.go:356] "Generic (PLEG): container finished" podID="7fd727e7-9b16-4ba3-aea7-771a1db8e913" containerID="efb87c517c021a045e24fafbc144a5e8f03f5f15f9fd4c99a28fb7ff67365a79" exitCode=0 Sep 11 14:14:23.379654 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:23.379281 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" event={"ID":"7fd727e7-9b16-4ba3-aea7-771a1db8e913","Type":"ContainerDied","Data":"efb87c517c021a045e24fafbc144a5e8f03f5f15f9fd4c99a28fb7ff67365a79"} Sep 11 14:14:23.379725 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:23.379711 2566 scope.go:117] "RemoveContainer" containerID="efb87c517c021a045e24fafbc144a5e8f03f5f15f9fd4c99a28fb7ff67365a79" Sep 11 14:14:23.380907 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:23.380871 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:14:23.380973 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:23.380957 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:14:23.381017 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:23.380995 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:14:24.387845 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:24.387813 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" event={"ID":"7fd727e7-9b16-4ba3-aea7-771a1db8e913","Type":"ContainerStarted","Data":"0fdd2d9b1ed8c16d4e1da7bcc3eed2aa7fe2ace1e75cab54e572ebcd3aa97ae5"} Sep 11 14:14:25.041130 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:25.041097 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:14:25.041439 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:25.041413 2566 patch_prober.go:28] interesting pod/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:14:25.041504 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:25.041477 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" podUID="7fd727e7-9b16-4ba3-aea7-771a1db8e913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 11 14:14:26.041052 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:26.041017 2566 patch_prober.go:28] interesting pod/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:14:26.041470 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:26.041102 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" podUID="7fd727e7-9b16-4ba3-aea7-771a1db8e913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 11 14:14:27.041281 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:27.041236 2566 patch_prober.go:28] interesting pod/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:14:27.041788 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:27.041303 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" podUID="7fd727e7-9b16-4ba3-aea7-771a1db8e913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 11 14:14:28.040538 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:28.040499 2566 patch_prober.go:28] interesting pod/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:14:28.040737 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:28.040562 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" podUID="7fd727e7-9b16-4ba3-aea7-771a1db8e913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 11 14:14:29.040704 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:29.040661 2566 patch_prober.go:28] interesting pod/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:14:29.041168 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:29.040731 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" podUID="7fd727e7-9b16-4ba3-aea7-771a1db8e913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 11 14:14:30.044283 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:30.044251 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:14:30.044762 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:30.044605 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:14:30.045343 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:30.045324 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:14:33.524059 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:33.524024 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv"] Sep 11 14:14:33.524476 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:33.524462 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="759456e0-a5a2-4d54-8938-a9e6730693f5" containerName="istio-proxy" Sep 11 14:14:33.524523 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:33.524478 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="759456e0-a5a2-4d54-8938-a9e6730693f5" containerName="istio-proxy" Sep 11 14:14:33.524558 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:33.524547 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="759456e0-a5a2-4d54-8938-a9e6730693f5" containerName="istio-proxy" Sep 11 14:14:33.528183 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:33.528162 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv" Sep 11 14:14:33.534440 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:33.534422 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:14:33.554322 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:33.554288 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv"] Sep 11 14:14:33.575392 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:33.575364 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db98d108-c313-4b15-b156-3dd53d015efb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv\" (UID: \"db98d108-c313-4b15-b156-3dd53d015efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv" Sep 11 14:14:33.676753 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:33.676705 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db98d108-c313-4b15-b156-3dd53d015efb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv\" (UID: \"db98d108-c313-4b15-b156-3dd53d015efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv" Sep 11 14:14:33.677378 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:33.677358 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db98d108-c313-4b15-b156-3dd53d015efb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv\" (UID: \"db98d108-c313-4b15-b156-3dd53d015efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv" Sep 11 14:14:33.838557 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:33.838473 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv" Sep 11 14:14:34.118576 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:34.118545 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv"] Sep 11 14:14:34.119908 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:14:34.119882 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb98d108_c313_4b15_b156_3dd53d015efb.slice/crio-846f50d490125fd4b1e84cd93c3ddc5ebc809196e0596f1dd235954ca674b84d WatchSource:0}: Error finding container 846f50d490125fd4b1e84cd93c3ddc5ebc809196e0596f1dd235954ca674b84d: Status 404 returned error can't find the container with id 846f50d490125fd4b1e84cd93c3ddc5ebc809196e0596f1dd235954ca674b84d Sep 11 14:14:34.463925 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:34.463884 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv" event={"ID":"db98d108-c313-4b15-b156-3dd53d015efb","Type":"ContainerStarted","Data":"70c2d47a495db435fc71a869933b1c83c12ebd7e1f9e3ed4969b728b6ba9dcb8"} Sep 11 14:14:34.464142 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:34.463949 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv" event={"ID":"db98d108-c313-4b15-b156-3dd53d015efb","Type":"ContainerStarted","Data":"846f50d490125fd4b1e84cd93c3ddc5ebc809196e0596f1dd235954ca674b84d"} Sep 11 14:14:34.488131 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:34.488066 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv" podStartSLOduration=1.4880507170000001 podStartE2EDuration="1.488050717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:14:34.483555544 +0000 UTC m=+1014.888536672" watchObservedRunningTime="2026-09-11 14:14:34.488050717 +0000 UTC m=+1014.893031894" Sep 11 14:14:35.472516 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:35.472483 2566 generic.go:356] "Generic (PLEG): container finished" podID="db98d108-c313-4b15-b156-3dd53d015efb" containerID="70c2d47a495db435fc71a869933b1c83c12ebd7e1f9e3ed4969b728b6ba9dcb8" exitCode=0 Sep 11 14:14:35.473013 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:35.472573 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv" event={"ID":"db98d108-c313-4b15-b156-3dd53d015efb","Type":"ContainerDied","Data":"70c2d47a495db435fc71a869933b1c83c12ebd7e1f9e3ed4969b728b6ba9dcb8"} Sep 11 14:14:36.662261 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:36.662233 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv" Sep 11 14:14:36.711502 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:36.711216 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db98d108-c313-4b15-b156-3dd53d015efb-maas-api-service-ca\") pod \"db98d108-c313-4b15-b156-3dd53d015efb\" (UID: \"db98d108-c313-4b15-b156-3dd53d015efb\") " Sep 11 14:14:36.711991 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:36.711869 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db98d108-c313-4b15-b156-3dd53d015efb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "db98d108-c313-4b15-b156-3dd53d015efb" (UID: "db98d108-c313-4b15-b156-3dd53d015efb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:14:36.712503 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:36.712480 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db98d108-c313-4b15-b156-3dd53d015efb-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:37.486891 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:37.486865 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv" Sep 11 14:14:37.487060 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:37.486905 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv" event={"ID":"db98d108-c313-4b15-b156-3dd53d015efb","Type":"ContainerDied","Data":"846f50d490125fd4b1e84cd93c3ddc5ebc809196e0596f1dd235954ca674b84d"} Sep 11 14:14:37.487060 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:37.486953 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="846f50d490125fd4b1e84cd93c3ddc5ebc809196e0596f1dd235954ca674b84d" Sep 11 14:14:45.857680 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:45.857649 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv"] Sep 11 14:14:45.863600 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:45.863571 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-59484b3d-nqljv"] Sep 11 14:14:45.991517 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:45.991488 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc"] Sep 11 14:14:45.991736 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:45.991700 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" podUID="7fd727e7-9b16-4ba3-aea7-771a1db8e913" containerName="istio-proxy" containerID="cri-o://0fdd2d9b1ed8c16d4e1da7bcc3eed2aa7fe2ace1e75cab54e572ebcd3aa97ae5" gracePeriod=30 Sep 11 14:14:46.103812 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:46.103768 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db98d108-c313-4b15-b156-3dd53d015efb" path="/var/lib/kubelet/pods/db98d108-c313-4b15-b156-3dd53d015efb/volumes" Sep 11 14:14:46.905449 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:46.905408 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh"] Sep 11 14:14:46.905923 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:46.905823 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="759456e0-a5a2-4d54-8938-a9e6730693f5" containerName="istio-proxy" Sep 11 14:14:46.905923 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:46.905833 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="759456e0-a5a2-4d54-8938-a9e6730693f5" containerName="istio-proxy" Sep 11 14:14:46.905923 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:46.905845 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db98d108-c313-4b15-b156-3dd53d015efb" containerName="revoke-keys" Sep 11 14:14:46.905923 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:46.905851 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="db98d108-c313-4b15-b156-3dd53d015efb" containerName="revoke-keys" Sep 11 14:14:46.905923 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:46.905924 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="db98d108-c313-4b15-b156-3dd53d015efb" containerName="revoke-keys" Sep 11 14:14:46.906209 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:46.905934 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="759456e0-a5a2-4d54-8938-a9e6730693f5" containerName="istio-proxy" Sep 11 14:14:46.908556 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:46.908536 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:46.911886 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:46.911860 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-65bba332-openshift-default-dockercfg-cbp45\"" Sep 11 14:14:46.940211 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:46.940185 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh"] Sep 11 14:14:47.015232 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.015196 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-credential-socket\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.015415 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.015243 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-podinfo\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.015415 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.015273 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-data\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.015415 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.015326 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppkv5\" (UniqueName: \"kubernetes.io/projected/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-kube-api-access-ppkv5\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.015415 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.015349 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-envoy\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.015415 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.015374 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-token\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.015415 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.015414 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-workload-certs\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.015641 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.015472 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istiod-ca-cert\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.015641 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.015489 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-workload-socket\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.116405 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.116363 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-credential-socket\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.116405 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.116408 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-podinfo\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.116657 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.116439 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-data\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.116657 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.116492 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppkv5\" (UniqueName: \"kubernetes.io/projected/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-kube-api-access-ppkv5\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.116657 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.116527 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-envoy\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.116657 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.116559 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-token\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.116657 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.116594 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-workload-certs\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.116929 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.116683 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istiod-ca-cert\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.116929 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.116708 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-workload-socket\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.116929 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.116825 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-credential-socket\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.116929 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.116866 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-data\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.117240 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.117031 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-workload-socket\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.117240 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.117182 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-workload-certs\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.117429 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.117412 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istiod-ca-cert\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.118893 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.118873 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-envoy\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.119178 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.119162 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-podinfo\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.176490 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.176414 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppkv5\" (UniqueName: \"kubernetes.io/projected/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-kube-api-access-ppkv5\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.221575 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.221550 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-token\") pod \"e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.519903 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.519863 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:47.854027 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.854000 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh"] Sep 11 14:14:47.855739 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:14:47.855711 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8d90ca_d946_44f9_87db_72ef40f4f7e8.slice/crio-cfd92945f060595b608298b0be410d945bff94cd4afebfbd9b119e0f880d8dc9 WatchSource:0}: Error finding container cfd92945f060595b608298b0be410d945bff94cd4afebfbd9b119e0f880d8dc9: Status 404 returned error can't find the container with id cfd92945f060595b608298b0be410d945bff94cd4afebfbd9b119e0f880d8dc9 Sep 11 14:14:47.858069 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.858041 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:14:47.858157 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.858120 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:14:47.858157 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:47.858149 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:14:48.564019 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:48.563977 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" event={"ID":"ad8d90ca-d946-44f9-87db-72ef40f4f7e8","Type":"ContainerStarted","Data":"4fa8bc9cc0d9a9ce59e2e16a8dd92afb8230dabc8952c9640a681db18e902f44"} Sep 11 14:14:48.564490 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:48.564035 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" event={"ID":"ad8d90ca-d946-44f9-87db-72ef40f4f7e8","Type":"ContainerStarted","Data":"cfd92945f060595b608298b0be410d945bff94cd4afebfbd9b119e0f880d8dc9"} Sep 11 14:14:48.602159 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:48.602073 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" podStartSLOduration=2.602056235 podStartE2EDuration="2.602056235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:14:48.600393339 +0000 UTC m=+1029.005374468" watchObservedRunningTime="2026-09-11 14:14:48.602056235 +0000 UTC m=+1029.007037363" Sep 11 14:14:49.520790 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:49.520725 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:49.526034 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:49.526007 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:49.570057 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:49.570025 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:49.570969 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:49.570949 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:14:51.253069 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.253043 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:14:51.358289 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358205 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istiod-ca-cert\") pod \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " Sep 11 14:14:51.358289 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358244 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-credential-socket\") pod \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " Sep 11 14:14:51.358289 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358271 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-podinfo\") pod \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " Sep 11 14:14:51.358584 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358324 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-workload-socket\") pod \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " Sep 11 14:14:51.358584 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358379 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-envoy\") pod \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " Sep 11 14:14:51.358584 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358400 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-token\") pod \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " Sep 11 14:14:51.358584 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358415 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-data\") pod \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " Sep 11 14:14:51.358584 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358442 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-workload-certs\") pod \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " Sep 11 14:14:51.358584 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358473 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpnqt\" (UniqueName: \"kubernetes.io/projected/7fd727e7-9b16-4ba3-aea7-771a1db8e913-kube-api-access-wpnqt\") pod \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\" (UID: \"7fd727e7-9b16-4ba3-aea7-771a1db8e913\") " Sep 11 14:14:51.358896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358601 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7fd727e7-9b16-4ba3-aea7-771a1db8e913" (UID: "7fd727e7-9b16-4ba3-aea7-771a1db8e913"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:14:51.358896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358639 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7fd727e7-9b16-4ba3-aea7-771a1db8e913" (UID: "7fd727e7-9b16-4ba3-aea7-771a1db8e913"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:14:51.358896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358853 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7fd727e7-9b16-4ba3-aea7-771a1db8e913" (UID: "7fd727e7-9b16-4ba3-aea7-771a1db8e913"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:14:51.358896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358865 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-data" (OuterVolumeSpecName: "istio-data") pod "7fd727e7-9b16-4ba3-aea7-771a1db8e913" (UID: "7fd727e7-9b16-4ba3-aea7-771a1db8e913"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:14:51.358896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358883 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:51.358896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358884 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7fd727e7-9b16-4ba3-aea7-771a1db8e913" (UID: "7fd727e7-9b16-4ba3-aea7-771a1db8e913"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:14:51.358896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.358903 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:51.361016 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.360988 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7fd727e7-9b16-4ba3-aea7-771a1db8e913" (UID: "7fd727e7-9b16-4ba3-aea7-771a1db8e913"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:14:51.361153 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.361076 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd727e7-9b16-4ba3-aea7-771a1db8e913-kube-api-access-wpnqt" (OuterVolumeSpecName: "kube-api-access-wpnqt") pod "7fd727e7-9b16-4ba3-aea7-771a1db8e913" (UID: "7fd727e7-9b16-4ba3-aea7-771a1db8e913"). InnerVolumeSpecName "kube-api-access-wpnqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:14:51.361153 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.361075 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7fd727e7-9b16-4ba3-aea7-771a1db8e913" (UID: "7fd727e7-9b16-4ba3-aea7-771a1db8e913"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:14:51.361314 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.361294 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-token" (OuterVolumeSpecName: "istio-token") pod "7fd727e7-9b16-4ba3-aea7-771a1db8e913" (UID: "7fd727e7-9b16-4ba3-aea7-771a1db8e913"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:14:51.460328 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.460293 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:51.460328 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.460322 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:51.460328 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.460330 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:51.460328 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.460338 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpnqt\" (UniqueName: \"kubernetes.io/projected/7fd727e7-9b16-4ba3-aea7-771a1db8e913-kube-api-access-wpnqt\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:51.460605 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.460348 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:51.460605 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.460356 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:51.460605 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.460364 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd727e7-9b16-4ba3-aea7-771a1db8e913-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:14:51.583582 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.583542 2566 generic.go:356] "Generic (PLEG): container finished" podID="7fd727e7-9b16-4ba3-aea7-771a1db8e913" containerID="0fdd2d9b1ed8c16d4e1da7bcc3eed2aa7fe2ace1e75cab54e572ebcd3aa97ae5" exitCode=0 Sep 11 14:14:51.583782 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.583598 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" Sep 11 14:14:51.583782 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.583621 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" event={"ID":"7fd727e7-9b16-4ba3-aea7-771a1db8e913","Type":"ContainerDied","Data":"0fdd2d9b1ed8c16d4e1da7bcc3eed2aa7fe2ace1e75cab54e572ebcd3aa97ae5"} Sep 11 14:14:51.583782 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.583678 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc" event={"ID":"7fd727e7-9b16-4ba3-aea7-771a1db8e913","Type":"ContainerDied","Data":"d60c1980aeacd0b90c043a52ee61ce53ceb0f86c6023a511123a25770b29e38d"} Sep 11 14:14:51.583782 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.583699 2566 scope.go:117] "RemoveContainer" containerID="0fdd2d9b1ed8c16d4e1da7bcc3eed2aa7fe2ace1e75cab54e572ebcd3aa97ae5" Sep 11 14:14:51.593873 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.593850 2566 scope.go:117] "RemoveContainer" containerID="efb87c517c021a045e24fafbc144a5e8f03f5f15f9fd4c99a28fb7ff67365a79" Sep 11 14:14:51.604761 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.604734 2566 scope.go:117] "RemoveContainer" containerID="0fdd2d9b1ed8c16d4e1da7bcc3eed2aa7fe2ace1e75cab54e572ebcd3aa97ae5" Sep 11 14:14:51.605136 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:14:51.605108 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fdd2d9b1ed8c16d4e1da7bcc3eed2aa7fe2ace1e75cab54e572ebcd3aa97ae5\": container with ID starting with 0fdd2d9b1ed8c16d4e1da7bcc3eed2aa7fe2ace1e75cab54e572ebcd3aa97ae5 not found: ID does not exist" containerID="0fdd2d9b1ed8c16d4e1da7bcc3eed2aa7fe2ace1e75cab54e572ebcd3aa97ae5" Sep 11 14:14:51.605245 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.605147 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fdd2d9b1ed8c16d4e1da7bcc3eed2aa7fe2ace1e75cab54e572ebcd3aa97ae5"} err="failed to get container status \"0fdd2d9b1ed8c16d4e1da7bcc3eed2aa7fe2ace1e75cab54e572ebcd3aa97ae5\": rpc error: code = NotFound desc = could not find container \"0fdd2d9b1ed8c16d4e1da7bcc3eed2aa7fe2ace1e75cab54e572ebcd3aa97ae5\": container with ID starting with 0fdd2d9b1ed8c16d4e1da7bcc3eed2aa7fe2ace1e75cab54e572ebcd3aa97ae5 not found: ID does not exist" Sep 11 14:14:51.605245 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.605168 2566 scope.go:117] "RemoveContainer" containerID="efb87c517c021a045e24fafbc144a5e8f03f5f15f9fd4c99a28fb7ff67365a79" Sep 11 14:14:51.605449 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:14:51.605427 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb87c517c021a045e24fafbc144a5e8f03f5f15f9fd4c99a28fb7ff67365a79\": container with ID starting with efb87c517c021a045e24fafbc144a5e8f03f5f15f9fd4c99a28fb7ff67365a79 not found: ID does not exist" containerID="efb87c517c021a045e24fafbc144a5e8f03f5f15f9fd4c99a28fb7ff67365a79" Sep 11 14:14:51.605498 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.605458 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb87c517c021a045e24fafbc144a5e8f03f5f15f9fd4c99a28fb7ff67365a79"} err="failed to get container status \"efb87c517c021a045e24fafbc144a5e8f03f5f15f9fd4c99a28fb7ff67365a79\": rpc error: code = NotFound desc = could not find container \"efb87c517c021a045e24fafbc144a5e8f03f5f15f9fd4c99a28fb7ff67365a79\": container with ID starting with efb87c517c021a045e24fafbc144a5e8f03f5f15f9fd4c99a28fb7ff67365a79 not found: ID does not exist" Sep 11 14:14:51.609406 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.609353 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc"] Sep 11 14:14:51.614012 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:51.613992 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-59484b3d-gw-openshift-default-5b58f7584-msjhc"] Sep 11 14:14:52.100747 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:14:52.100706 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fd727e7-9b16-4ba3-aea7-771a1db8e913" path="/var/lib/kubelet/pods/7fd727e7-9b16-4ba3-aea7-771a1db8e913/volumes" Sep 11 14:15:00.170194 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.170161 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be0c80-29818935-nmnb4"] Sep 11 14:15:00.170592 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.170579 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd727e7-9b16-4ba3-aea7-771a1db8e913" containerName="istio-proxy" Sep 11 14:15:00.170592 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.170593 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd727e7-9b16-4ba3-aea7-771a1db8e913" containerName="istio-proxy" Sep 11 14:15:00.170665 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.170604 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd727e7-9b16-4ba3-aea7-771a1db8e913" containerName="istio-proxy" Sep 11 14:15:00.170665 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.170610 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd727e7-9b16-4ba3-aea7-771a1db8e913" containerName="istio-proxy" Sep 11 14:15:00.170728 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.170679 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd727e7-9b16-4ba3-aea7-771a1db8e913" containerName="istio-proxy" Sep 11 14:15:00.170728 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.170688 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd727e7-9b16-4ba3-aea7-771a1db8e913" containerName="istio-proxy" Sep 11 14:15:00.173206 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.173188 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be0c80-29818935-nmnb4" Sep 11 14:15:00.179249 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.179217 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65bba332-29818935-xw2wb"] Sep 11 14:15:00.181159 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.181134 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:15:00.182244 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.182228 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65bba332-29818935-xw2wb" Sep 11 14:15:00.182816 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.182802 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818935-c99c9"] Sep 11 14:15:00.183799 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.183779 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be0c80-29818935-nmnb4" Sep 11 14:15:00.184916 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.184887 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-c99c9" Sep 11 14:15:00.189486 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.189469 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2a21d9-29818935-hnjd8"] Sep 11 14:15:00.192181 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.192159 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2a21d9-29818935-hnjd8" Sep 11 14:15:00.199566 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.194754 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65bba332-29818935-xw2wb" Sep 11 14:15:00.205144 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.205057 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be0c80-29818935-nmnb4"] Sep 11 14:15:00.210191 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.210164 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65bba332-29818935-xw2wb"] Sep 11 14:15:00.214588 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.214555 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818935-c99c9"] Sep 11 14:15:00.226419 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.226392 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2a21d9-29818935-hnjd8"] Sep 11 14:15:00.247026 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.246999 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-c99c9" Sep 11 14:15:00.257408 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.257357 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2a21d9-29818935-hnjd8" Sep 11 14:15:00.516011 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:15:00.515981 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfafad60_2828_4eed_a741_49f914583bef.slice/crio-23da8f41d48fe8240e2be99726eff2c1d88991c3b74e86364f13fc0ae6eab88e WatchSource:0}: Error finding container 23da8f41d48fe8240e2be99726eff2c1d88991c3b74e86364f13fc0ae6eab88e: Status 404 returned error can't find the container with id 23da8f41d48fe8240e2be99726eff2c1d88991c3b74e86364f13fc0ae6eab88e Sep 11 14:15:00.516918 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.516878 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be0c80-29818935-nmnb4"] Sep 11 14:15:00.544205 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.544172 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65bba332-29818935-xw2wb"] Sep 11 14:15:00.593350 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.593322 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2a21d9-29818935-hnjd8"] Sep 11 14:15:00.595339 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:15:00.595311 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c29f9f8_86b9_4a72_9647_090933f9d110.slice/crio-e852f0c7bc1e6ff324421d2b8381f645ce325bb24fa339919a489adb8c4bf051 WatchSource:0}: Error finding container e852f0c7bc1e6ff324421d2b8381f645ce325bb24fa339919a489adb8c4bf051: Status 404 returned error can't find the container with id e852f0c7bc1e6ff324421d2b8381f645ce325bb24fa339919a489adb8c4bf051 Sep 11 14:15:00.596661 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.596639 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818935-c99c9"] Sep 11 14:15:00.644956 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.644917 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65bba332-29818935-xw2wb" event={"ID":"cdc756bf-6b43-4397-bd0f-f92436233c51","Type":"ContainerStarted","Data":"647fb84b25fbdd60ef589e85ee6e568eb6b48308d2cd6f86fbf88fc9e15d8f6f"} Sep 11 14:15:00.646405 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.646377 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be0c80-29818935-nmnb4" event={"ID":"bfafad60-2828-4eed-a741-49f914583bef","Type":"ContainerStarted","Data":"23da8f41d48fe8240e2be99726eff2c1d88991c3b74e86364f13fc0ae6eab88e"} Sep 11 14:15:00.647702 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.647678 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2a21d9-29818935-hnjd8" event={"ID":"2c29f9f8-86b9-4a72-9647-090933f9d110","Type":"ContainerStarted","Data":"e852f0c7bc1e6ff324421d2b8381f645ce325bb24fa339919a489adb8c4bf051"} Sep 11 14:15:00.648969 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:00.648950 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-c99c9" event={"ID":"4c8551ad-387a-468c-9b5e-eb10a7df4b21","Type":"ContainerStarted","Data":"bc4d019b6e862ea19ee16ae10a16ef882f7fd5582a93040b0b7e2c7f54bd74f8"} Sep 11 14:15:01.656602 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:01.656559 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2a21d9-29818935-hnjd8" event={"ID":"2c29f9f8-86b9-4a72-9647-090933f9d110","Type":"ContainerStarted","Data":"0cecf60879aaa7ec3951a54717f6de04eb17a8a972de52244e28a4d7ca053f25"} Sep 11 14:15:01.658311 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:01.658279 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-c99c9" event={"ID":"4c8551ad-387a-468c-9b5e-eb10a7df4b21","Type":"ContainerStarted","Data":"7814039e6908cf0050e12fd32b5d4bfe8ab9ab6522125eb35b2e49485bd0fa4d"} Sep 11 14:15:01.659762 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:01.659739 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65bba332-29818935-xw2wb" event={"ID":"cdc756bf-6b43-4397-bd0f-f92436233c51","Type":"ContainerStarted","Data":"15220d652f552f9c23f2a8b3ac22b781679169a593612506ed42bef2a0caaa82"} Sep 11 14:15:01.661172 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:01.661147 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be0c80-29818935-nmnb4" event={"ID":"bfafad60-2828-4eed-a741-49f914583bef","Type":"ContainerStarted","Data":"087ee3c0d89494a29421de1c03e00db3672f9a49d3f9ad109d5ee3b29e94c062"} Sep 11 14:15:01.676177 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:01.676135 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2a21d9-29818935-hnjd8" podStartSLOduration=1.676119231 podStartE2EDuration="1.676119231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:01.673350969 +0000 UTC m=+1042.078332089" watchObservedRunningTime="2026-09-11 14:15:01.676119231 +0000 UTC m=+1042.081100394" Sep 11 14:15:01.695073 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:01.695027 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be0c80-29818935-nmnb4" podStartSLOduration=1.69501539 podStartE2EDuration="1.69501539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:01.693604503 +0000 UTC m=+1042.098585633" watchObservedRunningTime="2026-09-11 14:15:01.69501539 +0000 UTC m=+1042.099996518" Sep 11 14:15:01.712690 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:01.712642 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-c99c9" podStartSLOduration=1.712629873 podStartE2EDuration="1.712629873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:01.710777249 +0000 UTC m=+1042.115758388" watchObservedRunningTime="2026-09-11 14:15:01.712629873 +0000 UTC m=+1042.117611000" Sep 11 14:15:01.734772 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:01.734722 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65bba332-29818935-xw2wb" podStartSLOduration=1.734702436 podStartE2EDuration="1.734702436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:01.732485773 +0000 UTC m=+1042.137466902" watchObservedRunningTime="2026-09-11 14:15:01.734702436 +0000 UTC m=+1042.139683565" Sep 11 14:15:02.671205 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:02.671162 2566 generic.go:356] "Generic (PLEG): container finished" podID="4c8551ad-387a-468c-9b5e-eb10a7df4b21" containerID="7814039e6908cf0050e12fd32b5d4bfe8ab9ab6522125eb35b2e49485bd0fa4d" exitCode=0 Sep 11 14:15:02.671653 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:02.671247 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-c99c9" event={"ID":"4c8551ad-387a-468c-9b5e-eb10a7df4b21","Type":"ContainerDied","Data":"7814039e6908cf0050e12fd32b5d4bfe8ab9ab6522125eb35b2e49485bd0fa4d"} Sep 11 14:15:02.673184 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:02.673153 2566 generic.go:356] "Generic (PLEG): container finished" podID="cdc756bf-6b43-4397-bd0f-f92436233c51" containerID="15220d652f552f9c23f2a8b3ac22b781679169a593612506ed42bef2a0caaa82" exitCode=0 Sep 11 14:15:02.673322 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:02.673239 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65bba332-29818935-xw2wb" event={"ID":"cdc756bf-6b43-4397-bd0f-f92436233c51","Type":"ContainerDied","Data":"15220d652f552f9c23f2a8b3ac22b781679169a593612506ed42bef2a0caaa82"} Sep 11 14:15:02.674989 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:02.674959 2566 generic.go:356] "Generic (PLEG): container finished" podID="bfafad60-2828-4eed-a741-49f914583bef" containerID="087ee3c0d89494a29421de1c03e00db3672f9a49d3f9ad109d5ee3b29e94c062" exitCode=0 Sep 11 14:15:02.675131 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:02.675005 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be0c80-29818935-nmnb4" event={"ID":"bfafad60-2828-4eed-a741-49f914583bef","Type":"ContainerDied","Data":"087ee3c0d89494a29421de1c03e00db3672f9a49d3f9ad109d5ee3b29e94c062"} Sep 11 14:15:02.676617 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:02.676594 2566 generic.go:356] "Generic (PLEG): container finished" podID="2c29f9f8-86b9-4a72-9647-090933f9d110" containerID="0cecf60879aaa7ec3951a54717f6de04eb17a8a972de52244e28a4d7ca053f25" exitCode=0 Sep 11 14:15:02.676708 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:02.676674 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2a21d9-29818935-hnjd8" event={"ID":"2c29f9f8-86b9-4a72-9647-090933f9d110","Type":"ContainerDied","Data":"0cecf60879aaa7ec3951a54717f6de04eb17a8a972de52244e28a4d7ca053f25"} Sep 11 14:15:03.945811 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:03.944438 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2a21d9-29818935-hnjd8" Sep 11 14:15:03.948794 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:03.948713 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65bba332-29818935-xw2wb" Sep 11 14:15:03.981467 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:03.981395 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-c99c9" Sep 11 14:15:03.991892 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:03.991266 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be0c80-29818935-nmnb4" Sep 11 14:15:04.690243 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:04.690200 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be0c80-29818935-nmnb4" event={"ID":"bfafad60-2828-4eed-a741-49f914583bef","Type":"ContainerDied","Data":"23da8f41d48fe8240e2be99726eff2c1d88991c3b74e86364f13fc0ae6eab88e"} Sep 11 14:15:04.690243 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:04.690229 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be0c80-29818935-nmnb4" Sep 11 14:15:04.690512 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:04.690265 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23da8f41d48fe8240e2be99726eff2c1d88991c3b74e86364f13fc0ae6eab88e" Sep 11 14:15:04.691662 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:04.691640 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2a21d9-29818935-hnjd8" Sep 11 14:15:04.691848 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:04.691675 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2a21d9-29818935-hnjd8" event={"ID":"2c29f9f8-86b9-4a72-9647-090933f9d110","Type":"ContainerDied","Data":"e852f0c7bc1e6ff324421d2b8381f645ce325bb24fa339919a489adb8c4bf051"} Sep 11 14:15:04.691848 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:04.691712 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e852f0c7bc1e6ff324421d2b8381f645ce325bb24fa339919a489adb8c4bf051" Sep 11 14:15:04.693393 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:04.693370 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-c99c9" event={"ID":"4c8551ad-387a-468c-9b5e-eb10a7df4b21","Type":"ContainerDied","Data":"bc4d019b6e862ea19ee16ae10a16ef882f7fd5582a93040b0b7e2c7f54bd74f8"} Sep 11 14:15:04.693547 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:04.693535 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc4d019b6e862ea19ee16ae10a16ef882f7fd5582a93040b0b7e2c7f54bd74f8" Sep 11 14:15:04.693614 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:04.693409 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818935-c99c9" Sep 11 14:15:04.694967 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:04.694947 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65bba332-29818935-xw2wb" event={"ID":"cdc756bf-6b43-4397-bd0f-f92436233c51","Type":"ContainerDied","Data":"647fb84b25fbdd60ef589e85ee6e568eb6b48308d2cd6f86fbf88fc9e15d8f6f"} Sep 11 14:15:04.695060 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:04.694965 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65bba332-29818935-xw2wb" Sep 11 14:15:04.695060 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:04.694983 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="647fb84b25fbdd60ef589e85ee6e568eb6b48308d2cd6f86fbf88fc9e15d8f6f" Sep 11 14:15:11.752252 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:11.752219 2566 generic.go:356] "Generic (PLEG): container finished" podID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" containerID="4fa8bc9cc0d9a9ce59e2e16a8dd92afb8230dabc8952c9640a681db18e902f44" exitCode=0 Sep 11 14:15:11.752687 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:11.752301 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" event={"ID":"ad8d90ca-d946-44f9-87db-72ef40f4f7e8","Type":"ContainerDied","Data":"4fa8bc9cc0d9a9ce59e2e16a8dd92afb8230dabc8952c9640a681db18e902f44"} Sep 11 14:15:11.752734 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:11.752714 2566 scope.go:117] "RemoveContainer" containerID="4fa8bc9cc0d9a9ce59e2e16a8dd92afb8230dabc8952c9640a681db18e902f44" Sep 11 14:15:11.753809 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:11.753782 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:15:11.753907 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:11.753847 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:15:11.753907 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:11.753878 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:15:12.760579 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:12.760543 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" event={"ID":"ad8d90ca-d946-44f9-87db-72ef40f4f7e8","Type":"ContainerStarted","Data":"4f247308c92179c62d37aab22aeda069627dcc9611a8a006f5e25ac62f5bfd56"} Sep 11 14:15:13.520634 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:13.520584 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:15:13.520872 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:13.520762 2566 patch_prober.go:28] interesting pod/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 11 14:15:13.520872 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:13.520817 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" podUID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 11 14:15:14.521300 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:14.521261 2566 patch_prober.go:28] interesting pod/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 11 14:15:14.521881 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:14.521852 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" podUID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 11 14:15:15.520990 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.520951 2566 patch_prober.go:28] interesting pod/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 11 14:15:15.521347 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.521315 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" podUID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 11 14:15:15.821798 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.821702 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-65bba332-5smnz"] Sep 11 14:15:15.822722 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.822693 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfafad60-2828-4eed-a741-49f914583bef" containerName="cleanup" Sep 11 14:15:15.822888 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.822876 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfafad60-2828-4eed-a741-49f914583bef" containerName="cleanup" Sep 11 14:15:15.822989 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.822979 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdc756bf-6b43-4397-bd0f-f92436233c51" containerName="cleanup" Sep 11 14:15:15.823078 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.823070 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdc756bf-6b43-4397-bd0f-f92436233c51" containerName="cleanup" Sep 11 14:15:15.823202 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.823189 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c29f9f8-86b9-4a72-9647-090933f9d110" containerName="cleanup" Sep 11 14:15:15.823289 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.823280 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c29f9f8-86b9-4a72-9647-090933f9d110" containerName="cleanup" Sep 11 14:15:15.823397 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.823387 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c8551ad-387a-468c-9b5e-eb10a7df4b21" containerName="cleanup" Sep 11 14:15:15.823477 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.823466 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c8551ad-387a-468c-9b5e-eb10a7df4b21" containerName="cleanup" Sep 11 14:15:15.823666 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.823653 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c29f9f8-86b9-4a72-9647-090933f9d110" containerName="cleanup" Sep 11 14:15:15.823765 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.823755 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdc756bf-6b43-4397-bd0f-f92436233c51" containerName="cleanup" Sep 11 14:15:15.840105 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.840056 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfafad60-2828-4eed-a741-49f914583bef" containerName="cleanup" Sep 11 14:15:15.840347 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.840321 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c8551ad-387a-468c-9b5e-eb10a7df4b21" containerName="cleanup" Sep 11 14:15:15.844128 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.844102 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-65bba332-5smnz"] Sep 11 14:15:15.844296 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.844256 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-65bba332-5smnz" Sep 11 14:15:15.848211 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:15.848188 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:15:16.017862 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:16.017819 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1429ad02-32c3-4973-a486-81491ef39828-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-65bba332-5smnz\" (UID: \"1429ad02-32c3-4973-a486-81491ef39828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-65bba332-5smnz" Sep 11 14:15:16.120139 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:16.119020 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1429ad02-32c3-4973-a486-81491ef39828-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-65bba332-5smnz\" (UID: \"1429ad02-32c3-4973-a486-81491ef39828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-65bba332-5smnz" Sep 11 14:15:16.120139 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:16.120012 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1429ad02-32c3-4973-a486-81491ef39828-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-65bba332-5smnz\" (UID: \"1429ad02-32c3-4973-a486-81491ef39828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-65bba332-5smnz" Sep 11 14:15:16.159726 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:16.159696 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-65bba332-5smnz" Sep 11 14:15:16.485425 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:16.485398 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-65bba332-5smnz"] Sep 11 14:15:16.486683 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:15:16.486654 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1429ad02_32c3_4973_a486_81491ef39828.slice/crio-6dbe7197ef0a53edefb731a318370e6e3ac537598aa7223b4a1498b72bd2ea0c WatchSource:0}: Error finding container 6dbe7197ef0a53edefb731a318370e6e3ac537598aa7223b4a1498b72bd2ea0c: Status 404 returned error can't find the container with id 6dbe7197ef0a53edefb731a318370e6e3ac537598aa7223b4a1498b72bd2ea0c Sep 11 14:15:16.523170 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:16.520459 2566 patch_prober.go:28] interesting pod/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 11 14:15:16.523170 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:16.520520 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" podUID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 11 14:15:16.788713 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:16.788631 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-65bba332-5smnz" event={"ID":"1429ad02-32c3-4973-a486-81491ef39828","Type":"ContainerStarted","Data":"6dbe7197ef0a53edefb731a318370e6e3ac537598aa7223b4a1498b72bd2ea0c"} Sep 11 14:15:17.521570 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:17.521117 2566 patch_prober.go:28] interesting pod/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 11 14:15:17.521570 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:17.521185 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" podUID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 11 14:15:17.797048 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:17.796965 2566 generic.go:356] "Generic (PLEG): container finished" podID="1429ad02-32c3-4973-a486-81491ef39828" containerID="7a56c43d3fc420ca52309f7c542e4bc536167560dd48cd2f5c047289e15895c7" exitCode=0 Sep 11 14:15:17.797559 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:17.797050 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-65bba332-5smnz" event={"ID":"1429ad02-32c3-4973-a486-81491ef39828","Type":"ContainerDied","Data":"7a56c43d3fc420ca52309f7c542e4bc536167560dd48cd2f5c047289e15895c7"} Sep 11 14:15:18.521354 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:18.521307 2566 patch_prober.go:28] interesting pod/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 11 14:15:18.521552 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:18.521407 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" podUID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 11 14:15:18.953538 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:18.953515 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-65bba332-5smnz" Sep 11 14:15:19.054509 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:19.054477 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1429ad02-32c3-4973-a486-81491ef39828-maas-api-service-ca\") pod \"1429ad02-32c3-4973-a486-81491ef39828\" (UID: \"1429ad02-32c3-4973-a486-81491ef39828\") " Sep 11 14:15:19.054833 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:19.054809 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1429ad02-32c3-4973-a486-81491ef39828-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1429ad02-32c3-4973-a486-81491ef39828" (UID: "1429ad02-32c3-4973-a486-81491ef39828"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:15:19.155755 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:19.155669 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1429ad02-32c3-4973-a486-81491ef39828-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:15:19.525118 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:19.525061 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:15:19.525601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:19.525579 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:15:19.526297 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:19.526279 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:15:19.814539 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:19.814460 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-65bba332-5smnz" Sep 11 14:15:19.814539 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:19.814496 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-65bba332-5smnz" event={"ID":"1429ad02-32c3-4973-a486-81491ef39828","Type":"ContainerDied","Data":"6dbe7197ef0a53edefb731a318370e6e3ac537598aa7223b4a1498b72bd2ea0c"} Sep 11 14:15:19.814721 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:19.814541 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dbe7197ef0a53edefb731a318370e6e3ac537598aa7223b4a1498b72bd2ea0c" Sep 11 14:15:21.740297 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:21.740263 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65bba332-29818935-xw2wb"] Sep 11 14:15:21.746770 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:21.746744 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-65bba332-29818935-xw2wb"] Sep 11 14:15:22.028233 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:22.028145 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-65bba332-5smnz"] Sep 11 14:15:22.034124 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:22.034095 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-65bba332-5smnz"] Sep 11 14:15:22.099918 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:22.099881 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1429ad02-32c3-4973-a486-81491ef39828" path="/var/lib/kubelet/pods/1429ad02-32c3-4973-a486-81491ef39828/volumes" Sep 11 14:15:22.100395 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:22.100370 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdc756bf-6b43-4397-bd0f-f92436233c51" path="/var/lib/kubelet/pods/cdc756bf-6b43-4397-bd0f-f92436233c51/volumes" Sep 11 14:15:23.325827 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:23.325794 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh"] Sep 11 14:15:23.326223 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:23.326034 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" podUID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" containerName="istio-proxy" containerID="cri-o://4f247308c92179c62d37aab22aeda069627dcc9611a8a006f5e25ac62f5bfd56" gracePeriod=30 Sep 11 14:15:24.098635 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.098604 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84"] Sep 11 14:15:24.099233 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.099211 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1429ad02-32c3-4973-a486-81491ef39828" containerName="revoke-keys" Sep 11 14:15:24.099233 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.099233 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1429ad02-32c3-4973-a486-81491ef39828" containerName="revoke-keys" Sep 11 14:15:24.099410 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.099392 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1429ad02-32c3-4973-a486-81491ef39828" containerName="revoke-keys" Sep 11 14:15:24.103357 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.103338 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.106567 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.106548 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d609a19a-openshift-default-dockercfg-8p5df\"" Sep 11 14:15:24.115931 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.115910 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84"] Sep 11 14:15:24.229236 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.229199 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-token\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.229460 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.229314 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62w2g\" (UniqueName: \"kubernetes.io/projected/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-kube-api-access-62w2g\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.229460 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.229373 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-data\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.229460 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.229416 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-podinfo\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.229460 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.229439 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istiod-ca-cert\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.229716 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.229488 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-workload-socket\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.229716 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.229511 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-envoy\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.229716 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.229556 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-workload-certs\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.229716 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.229596 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-credential-socket\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.330635 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.330597 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62w2g\" (UniqueName: \"kubernetes.io/projected/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-kube-api-access-62w2g\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.331063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.330642 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-data\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.331063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.330757 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-podinfo\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.331063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.330786 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istiod-ca-cert\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.331063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.330823 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-workload-socket\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.331063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.330849 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-envoy\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.331063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.330891 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-workload-certs\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.331063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.330935 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-credential-socket\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.331063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.331031 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-data\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.331063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.331035 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-token\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.331577 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.331209 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-workload-socket\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.331577 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.331368 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-workload-certs\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.331577 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.331398 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-credential-socket\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.331791 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.331768 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istiod-ca-cert\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.333261 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.333243 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-envoy\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.333552 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.333534 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-podinfo\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.341393 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.341362 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-token\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.341806 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.341785 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62w2g\" (UniqueName: \"kubernetes.io/projected/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-kube-api-access-62w2g\") pod \"e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.414753 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.414666 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:24.704932 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.704903 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84"] Sep 11 14:15:24.706353 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:15:24.706312 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970e6b7c_5827_4b2a_95cc_fe78e73b5b40.slice/crio-9410c88552b4d8439084abd3c07bbfd72b8b79685b13cbb602a29a6ae22a1a56 WatchSource:0}: Error finding container 9410c88552b4d8439084abd3c07bbfd72b8b79685b13cbb602a29a6ae22a1a56: Status 404 returned error can't find the container with id 9410c88552b4d8439084abd3c07bbfd72b8b79685b13cbb602a29a6ae22a1a56 Sep 11 14:15:24.708702 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.708670 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:15:24.708797 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.708745 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:15:24.708797 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.708772 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:15:24.862172 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.862132 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" event={"ID":"970e6b7c-5827-4b2a-95cc-fe78e73b5b40","Type":"ContainerStarted","Data":"507050beae6f498b39b6fa090128a96273671d1739e37543f233f42ab74095ed"} Sep 11 14:15:24.862342 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.862203 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" event={"ID":"970e6b7c-5827-4b2a-95cc-fe78e73b5b40","Type":"ContainerStarted","Data":"9410c88552b4d8439084abd3c07bbfd72b8b79685b13cbb602a29a6ae22a1a56"} Sep 11 14:15:24.923744 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:24.923695 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" podStartSLOduration=0.923676969 podStartE2EDuration="923.676969ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:24.921641517 +0000 UTC m=+1065.326622647" watchObservedRunningTime="2026-09-11 14:15:24.923676969 +0000 UTC m=+1065.328658096" Sep 11 14:15:25.414996 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:25.414962 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:25.420188 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:25.420166 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:25.868708 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:25.868674 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:25.869664 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:25.869644 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:28.623065 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.622708 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:15:28.681518 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.681484 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-workload-certs\") pod \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " Sep 11 14:15:28.681682 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.681568 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-podinfo\") pod \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " Sep 11 14:15:28.681682 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.681600 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-credential-socket\") pod \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " Sep 11 14:15:28.681682 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.681617 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-workload-socket\") pod \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " Sep 11 14:15:28.681882 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.681721 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ad8d90ca-d946-44f9-87db-72ef40f4f7e8" (UID: "ad8d90ca-d946-44f9-87db-72ef40f4f7e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:15:28.681882 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.681775 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-envoy\") pod \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " Sep 11 14:15:28.681882 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.681823 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istiod-ca-cert\") pod \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " Sep 11 14:15:28.681882 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.681862 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ad8d90ca-d946-44f9-87db-72ef40f4f7e8" (UID: "ad8d90ca-d946-44f9-87db-72ef40f4f7e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:15:28.682135 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.681883 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-data\") pod \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " Sep 11 14:15:28.682135 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.681887 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ad8d90ca-d946-44f9-87db-72ef40f4f7e8" (UID: "ad8d90ca-d946-44f9-87db-72ef40f4f7e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:15:28.682135 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.681943 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppkv5\" (UniqueName: \"kubernetes.io/projected/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-kube-api-access-ppkv5\") pod \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " Sep 11 14:15:28.682135 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.681973 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-token\") pod \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\" (UID: \"ad8d90ca-d946-44f9-87db-72ef40f4f7e8\") " Sep 11 14:15:28.682135 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.682039 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-data" (OuterVolumeSpecName: "istio-data") pod "ad8d90ca-d946-44f9-87db-72ef40f4f7e8" (UID: "ad8d90ca-d946-44f9-87db-72ef40f4f7e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:15:28.682400 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.682143 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ad8d90ca-d946-44f9-87db-72ef40f4f7e8" (UID: "ad8d90ca-d946-44f9-87db-72ef40f4f7e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:15:28.682400 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.682358 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:15:28.682400 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.682380 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:15:28.682400 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.682393 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:15:28.682781 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.682406 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:15:28.682781 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.682420 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:15:28.684271 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.684244 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ad8d90ca-d946-44f9-87db-72ef40f4f7e8" (UID: "ad8d90ca-d946-44f9-87db-72ef40f4f7e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:15:28.684435 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.684410 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ad8d90ca-d946-44f9-87db-72ef40f4f7e8" (UID: "ad8d90ca-d946-44f9-87db-72ef40f4f7e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:15:28.684775 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.684751 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-token" (OuterVolumeSpecName: "istio-token") pod "ad8d90ca-d946-44f9-87db-72ef40f4f7e8" (UID: "ad8d90ca-d946-44f9-87db-72ef40f4f7e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:15:28.684821 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.684756 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-kube-api-access-ppkv5" (OuterVolumeSpecName: "kube-api-access-ppkv5") pod "ad8d90ca-d946-44f9-87db-72ef40f4f7e8" (UID: "ad8d90ca-d946-44f9-87db-72ef40f4f7e8"). InnerVolumeSpecName "kube-api-access-ppkv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:15:28.783639 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.783606 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:15:28.783639 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.783634 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppkv5\" (UniqueName: \"kubernetes.io/projected/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-kube-api-access-ppkv5\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:15:28.783639 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.783643 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:15:28.783880 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.783653 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad8d90ca-d946-44f9-87db-72ef40f4f7e8-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:15:28.888747 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.888653 2566 generic.go:356] "Generic (PLEG): container finished" podID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" containerID="4f247308c92179c62d37aab22aeda069627dcc9611a8a006f5e25ac62f5bfd56" exitCode=0 Sep 11 14:15:28.888747 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.888706 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" Sep 11 14:15:28.888747 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.888735 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" event={"ID":"ad8d90ca-d946-44f9-87db-72ef40f4f7e8","Type":"ContainerDied","Data":"4f247308c92179c62d37aab22aeda069627dcc9611a8a006f5e25ac62f5bfd56"} Sep 11 14:15:28.889032 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.888788 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh" event={"ID":"ad8d90ca-d946-44f9-87db-72ef40f4f7e8","Type":"ContainerDied","Data":"cfd92945f060595b608298b0be410d945bff94cd4afebfbd9b119e0f880d8dc9"} Sep 11 14:15:28.889032 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.888809 2566 scope.go:117] "RemoveContainer" containerID="4f247308c92179c62d37aab22aeda069627dcc9611a8a006f5e25ac62f5bfd56" Sep 11 14:15:28.900396 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.900375 2566 scope.go:117] "RemoveContainer" containerID="4fa8bc9cc0d9a9ce59e2e16a8dd92afb8230dabc8952c9640a681db18e902f44" Sep 11 14:15:28.910200 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.910180 2566 scope.go:117] "RemoveContainer" containerID="4f247308c92179c62d37aab22aeda069627dcc9611a8a006f5e25ac62f5bfd56" Sep 11 14:15:28.910463 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:15:28.910447 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f247308c92179c62d37aab22aeda069627dcc9611a8a006f5e25ac62f5bfd56\": container with ID starting with 4f247308c92179c62d37aab22aeda069627dcc9611a8a006f5e25ac62f5bfd56 not found: ID does not exist" containerID="4f247308c92179c62d37aab22aeda069627dcc9611a8a006f5e25ac62f5bfd56" Sep 11 14:15:28.910517 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.910472 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f247308c92179c62d37aab22aeda069627dcc9611a8a006f5e25ac62f5bfd56"} err="failed to get container status \"4f247308c92179c62d37aab22aeda069627dcc9611a8a006f5e25ac62f5bfd56\": rpc error: code = NotFound desc = could not find container \"4f247308c92179c62d37aab22aeda069627dcc9611a8a006f5e25ac62f5bfd56\": container with ID starting with 4f247308c92179c62d37aab22aeda069627dcc9611a8a006f5e25ac62f5bfd56 not found: ID does not exist" Sep 11 14:15:28.910517 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.910488 2566 scope.go:117] "RemoveContainer" containerID="4fa8bc9cc0d9a9ce59e2e16a8dd92afb8230dabc8952c9640a681db18e902f44" Sep 11 14:15:28.910753 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:15:28.910735 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa8bc9cc0d9a9ce59e2e16a8dd92afb8230dabc8952c9640a681db18e902f44\": container with ID starting with 4fa8bc9cc0d9a9ce59e2e16a8dd92afb8230dabc8952c9640a681db18e902f44 not found: ID does not exist" containerID="4fa8bc9cc0d9a9ce59e2e16a8dd92afb8230dabc8952c9640a681db18e902f44" Sep 11 14:15:28.910792 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.910759 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa8bc9cc0d9a9ce59e2e16a8dd92afb8230dabc8952c9640a681db18e902f44"} err="failed to get container status \"4fa8bc9cc0d9a9ce59e2e16a8dd92afb8230dabc8952c9640a681db18e902f44\": rpc error: code = NotFound desc = could not find container \"4fa8bc9cc0d9a9ce59e2e16a8dd92afb8230dabc8952c9640a681db18e902f44\": container with ID starting with 4fa8bc9cc0d9a9ce59e2e16a8dd92afb8230dabc8952c9640a681db18e902f44 not found: ID does not exist" Sep 11 14:15:28.922610 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.922584 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh"] Sep 11 14:15:28.927630 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:28.927609 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-65bba332-openshift-default-588bffbff6-sxdjh"] Sep 11 14:15:30.100566 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:30.100524 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" path="/var/lib/kubelet/pods/ad8d90ca-d946-44f9-87db-72ef40f4f7e8/volumes" Sep 11 14:15:50.032496 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:50.032461 2566 generic.go:356] "Generic (PLEG): container finished" podID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" containerID="507050beae6f498b39b6fa090128a96273671d1739e37543f233f42ab74095ed" exitCode=0 Sep 11 14:15:50.032933 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:50.032521 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" event={"ID":"970e6b7c-5827-4b2a-95cc-fe78e73b5b40","Type":"ContainerDied","Data":"507050beae6f498b39b6fa090128a96273671d1739e37543f233f42ab74095ed"} Sep 11 14:15:50.032933 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:50.032915 2566 scope.go:117] "RemoveContainer" containerID="507050beae6f498b39b6fa090128a96273671d1739e37543f233f42ab74095ed" Sep 11 14:15:50.034065 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:50.034030 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:15:50.034214 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:50.034120 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:15:50.034214 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:50.034153 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:15:51.040508 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.040470 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" event={"ID":"970e6b7c-5827-4b2a-95cc-fe78e73b5b40","Type":"ContainerStarted","Data":"d667580ed6841ac9b721941fd717bd8b2fd8c10b55f6e1c3b27abdb006588cde"} Sep 11 14:15:51.188538 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.188505 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn"] Sep 11 14:15:51.188962 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.188947 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" containerName="istio-proxy" Sep 11 14:15:51.188962 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.188962 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" containerName="istio-proxy" Sep 11 14:15:51.189049 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.188980 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" containerName="istio-proxy" Sep 11 14:15:51.189049 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.188986 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" containerName="istio-proxy" Sep 11 14:15:51.189134 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.189056 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" containerName="istio-proxy" Sep 11 14:15:51.191429 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.191407 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn" Sep 11 14:15:51.196722 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.196702 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:15:51.209535 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.209510 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn"] Sep 11 14:15:51.218142 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.218116 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/856c4ec2-0156-4d6b-8a16-b99e081bb250-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn\" (UID: \"856c4ec2-0156-4d6b-8a16-b99e081bb250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn" Sep 11 14:15:51.319333 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.319250 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/856c4ec2-0156-4d6b-8a16-b99e081bb250-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn\" (UID: \"856c4ec2-0156-4d6b-8a16-b99e081bb250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn" Sep 11 14:15:51.319838 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.319820 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/856c4ec2-0156-4d6b-8a16-b99e081bb250-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn\" (UID: \"856c4ec2-0156-4d6b-8a16-b99e081bb250\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn" Sep 11 14:15:51.414881 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.414836 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:51.415721 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.415695 2566 patch_prober.go:28] interesting pod/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:15:51.415844 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.415760 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" podUID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 11 14:15:51.503548 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.503507 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn" Sep 11 14:15:51.823197 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:51.823158 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn"] Sep 11 14:15:51.826858 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:15:51.826816 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856c4ec2_0156_4d6b_8a16_b99e081bb250.slice/crio-d384f259476957106f1737729efa7fad2bbba34720616cae6aeb2003e8dca7db WatchSource:0}: Error finding container d384f259476957106f1737729efa7fad2bbba34720616cae6aeb2003e8dca7db: Status 404 returned error can't find the container with id d384f259476957106f1737729efa7fad2bbba34720616cae6aeb2003e8dca7db Sep 11 14:15:52.048435 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:52.048400 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn" event={"ID":"856c4ec2-0156-4d6b-8a16-b99e081bb250","Type":"ContainerStarted","Data":"d384f259476957106f1737729efa7fad2bbba34720616cae6aeb2003e8dca7db"} Sep 11 14:15:52.415731 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:52.415202 2566 patch_prober.go:28] interesting pod/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:15:52.415731 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:52.415270 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" podUID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 11 14:15:53.056298 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:53.056208 2566 generic.go:356] "Generic (PLEG): container finished" podID="856c4ec2-0156-4d6b-8a16-b99e081bb250" containerID="478d354fec8a2376894b376acd1204eb3ca23dcc9bfd895049fd4cdfea490bff" exitCode=0 Sep 11 14:15:53.056726 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:53.056524 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn" event={"ID":"856c4ec2-0156-4d6b-8a16-b99e081bb250","Type":"ContainerDied","Data":"478d354fec8a2376894b376acd1204eb3ca23dcc9bfd895049fd4cdfea490bff"} Sep 11 14:15:53.415529 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:53.415429 2566 patch_prober.go:28] interesting pod/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:15:53.415788 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:53.415759 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" podUID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 11 14:15:53.982135 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:53.982101 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc"] Sep 11 14:15:53.982767 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:53.982747 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8d90ca-d946-44f9-87db-72ef40f4f7e8" containerName="istio-proxy" Sep 11 14:15:53.984769 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:53.984750 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc" Sep 11 14:15:54.024029 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:54.022433 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc"] Sep 11 14:15:54.046433 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:54.046395 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f8619e-6674-4b0d-ba30-437b90e382a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc\" (UID: \"03f8619e-6674-4b0d-ba30-437b90e382a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc" Sep 11 14:15:54.147051 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:54.147017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f8619e-6674-4b0d-ba30-437b90e382a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc\" (UID: \"03f8619e-6674-4b0d-ba30-437b90e382a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc" Sep 11 14:15:54.147851 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:54.147824 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f8619e-6674-4b0d-ba30-437b90e382a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc\" (UID: \"03f8619e-6674-4b0d-ba30-437b90e382a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc" Sep 11 14:15:54.243696 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:54.243638 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn" Sep 11 14:15:54.294630 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:54.294594 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc" Sep 11 14:15:54.349033 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:54.349004 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/856c4ec2-0156-4d6b-8a16-b99e081bb250-maas-api-service-ca\") pod \"856c4ec2-0156-4d6b-8a16-b99e081bb250\" (UID: \"856c4ec2-0156-4d6b-8a16-b99e081bb250\") " Sep 11 14:15:54.349373 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:54.349350 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856c4ec2-0156-4d6b-8a16-b99e081bb250-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "856c4ec2-0156-4d6b-8a16-b99e081bb250" (UID: "856c4ec2-0156-4d6b-8a16-b99e081bb250"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:15:54.421109 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:54.420545 2566 patch_prober.go:28] interesting pod/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:15:54.421109 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:54.420612 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" podUID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 11 14:15:54.450137 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:54.450077 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/856c4ec2-0156-4d6b-8a16-b99e081bb250-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:15:54.615899 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:54.615860 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc"] Sep 11 14:15:55.072012 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:55.071978 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn" Sep 11 14:15:55.072236 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:55.072104 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn" event={"ID":"856c4ec2-0156-4d6b-8a16-b99e081bb250","Type":"ContainerDied","Data":"d384f259476957106f1737729efa7fad2bbba34720616cae6aeb2003e8dca7db"} Sep 11 14:15:55.072236 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:55.072172 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d384f259476957106f1737729efa7fad2bbba34720616cae6aeb2003e8dca7db" Sep 11 14:15:55.073393 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:55.073369 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc" event={"ID":"03f8619e-6674-4b0d-ba30-437b90e382a8","Type":"ContainerStarted","Data":"90a21d8d74aa73198a1cd53f54b905b35eb6caaa1f41b9e6f1eecfb001832376"} Sep 11 14:15:55.416072 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:55.416026 2566 patch_prober.go:28] interesting pod/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:15:55.416612 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:55.416127 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" podUID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 11 14:15:55.868613 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:55.868580 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:56.082371 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:56.082067 2566 generic.go:356] "Generic (PLEG): container finished" podID="03f8619e-6674-4b0d-ba30-437b90e382a8" containerID="7eb966c1c1d6fb7a1608d37255acb8bc88fd6fa8e854d28701ef1bf14741b6d2" exitCode=0 Sep 11 14:15:56.082557 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:56.082141 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc" event={"ID":"03f8619e-6674-4b0d-ba30-437b90e382a8","Type":"ContainerDied","Data":"7eb966c1c1d6fb7a1608d37255acb8bc88fd6fa8e854d28701ef1bf14741b6d2"} Sep 11 14:15:56.416119 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:56.415996 2566 patch_prober.go:28] interesting pod/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:15:56.416119 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:56.416064 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" podUID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 11 14:15:57.272815 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:57.272791 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc" Sep 11 14:15:57.380307 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:57.380207 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f8619e-6674-4b0d-ba30-437b90e382a8-maas-api-service-ca\") pod \"03f8619e-6674-4b0d-ba30-437b90e382a8\" (UID: \"03f8619e-6674-4b0d-ba30-437b90e382a8\") " Sep 11 14:15:57.380601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:57.380574 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f8619e-6674-4b0d-ba30-437b90e382a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "03f8619e-6674-4b0d-ba30-437b90e382a8" (UID: "03f8619e-6674-4b0d-ba30-437b90e382a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:15:57.419415 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:57.419385 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:57.420574 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:57.420553 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:15:57.481909 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:57.481874 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f8619e-6674-4b0d-ba30-437b90e382a8-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:15:58.097466 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:58.097440 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc" Sep 11 14:15:58.099876 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:58.099847 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc" event={"ID":"03f8619e-6674-4b0d-ba30-437b90e382a8","Type":"ContainerDied","Data":"90a21d8d74aa73198a1cd53f54b905b35eb6caaa1f41b9e6f1eecfb001832376"} Sep 11 14:15:58.100006 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:15:58.099899 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a21d8d74aa73198a1cd53f54b905b35eb6caaa1f41b9e6f1eecfb001832376" Sep 11 14:16:00.009535 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:00.009491 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn"] Sep 11 14:16:00.012232 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:00.012210 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d609a19a-kpmrn"] Sep 11 14:16:00.102904 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:00.102865 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="856c4ec2-0156-4d6b-8a16-b99e081bb250" path="/var/lib/kubelet/pods/856c4ec2-0156-4d6b-8a16-b99e081bb250/volumes" Sep 11 14:16:00.210178 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:00.210142 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84"] Sep 11 14:16:00.210368 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:00.210323 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" podUID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" containerName="istio-proxy" containerID="cri-o://d667580ed6841ac9b721941fd717bd8b2fd8c10b55f6e1c3b27abdb006588cde" gracePeriod=30 Sep 11 14:16:04.291362 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:04.291330 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2a21d9-29818935-hnjd8"] Sep 11 14:16:04.298771 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:04.298736 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-2a21d9-29818935-hnjd8"] Sep 11 14:16:04.785305 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:04.785271 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc"] Sep 11 14:16:04.791043 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:04.791017 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-2a21d9-w2dwc"] Sep 11 14:16:05.477220 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.477197 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:16:05.565879 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.565849 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-envoy\") pod \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " Sep 11 14:16:05.565879 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.565882 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-workload-certs\") pod \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " Sep 11 14:16:05.566137 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.565922 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istiod-ca-cert\") pod \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " Sep 11 14:16:05.566137 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.565940 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-data\") pod \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " Sep 11 14:16:05.566137 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.565977 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-token\") pod \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " Sep 11 14:16:05.566137 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.565998 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-podinfo\") pod \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " Sep 11 14:16:05.566137 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.566040 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-credential-socket\") pod \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " Sep 11 14:16:05.566137 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.566132 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62w2g\" (UniqueName: \"kubernetes.io/projected/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-kube-api-access-62w2g\") pod \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " Sep 11 14:16:05.566459 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.566187 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-workload-socket\") pod \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\" (UID: \"970e6b7c-5827-4b2a-95cc-fe78e73b5b40\") " Sep 11 14:16:05.566459 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.566317 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-data" (OuterVolumeSpecName: "istio-data") pod "970e6b7c-5827-4b2a-95cc-fe78e73b5b40" (UID: "970e6b7c-5827-4b2a-95cc-fe78e73b5b40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:16:05.566459 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.566333 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "970e6b7c-5827-4b2a-95cc-fe78e73b5b40" (UID: "970e6b7c-5827-4b2a-95cc-fe78e73b5b40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:16:05.566459 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.566406 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "970e6b7c-5827-4b2a-95cc-fe78e73b5b40" (UID: "970e6b7c-5827-4b2a-95cc-fe78e73b5b40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:16:05.566672 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.566541 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "970e6b7c-5827-4b2a-95cc-fe78e73b5b40" (UID: "970e6b7c-5827-4b2a-95cc-fe78e73b5b40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:16:05.566672 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.566649 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "970e6b7c-5827-4b2a-95cc-fe78e73b5b40" (UID: "970e6b7c-5827-4b2a-95cc-fe78e73b5b40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:16:05.566672 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.566663 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:16:05.566996 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.566680 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:16:05.566996 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.566692 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:16:05.566996 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.566702 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:16:05.568185 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.568164 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "970e6b7c-5827-4b2a-95cc-fe78e73b5b40" (UID: "970e6b7c-5827-4b2a-95cc-fe78e73b5b40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:16:05.568447 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.568426 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-kube-api-access-62w2g" (OuterVolumeSpecName: "kube-api-access-62w2g") pod "970e6b7c-5827-4b2a-95cc-fe78e73b5b40" (UID: "970e6b7c-5827-4b2a-95cc-fe78e73b5b40"). InnerVolumeSpecName "kube-api-access-62w2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:16:05.568500 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.568481 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-token" (OuterVolumeSpecName: "istio-token") pod "970e6b7c-5827-4b2a-95cc-fe78e73b5b40" (UID: "970e6b7c-5827-4b2a-95cc-fe78e73b5b40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:16:05.568558 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.568544 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "970e6b7c-5827-4b2a-95cc-fe78e73b5b40" (UID: "970e6b7c-5827-4b2a-95cc-fe78e73b5b40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:16:05.667461 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.667426 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:16:05.667461 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.667454 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:16:05.667461 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.667463 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:16:05.667708 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.667473 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:16:05.667708 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:05.667483 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62w2g\" (UniqueName: \"kubernetes.io/projected/970e6b7c-5827-4b2a-95cc-fe78e73b5b40-kube-api-access-62w2g\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:16:06.098801 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:06.098766 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03f8619e-6674-4b0d-ba30-437b90e382a8" path="/var/lib/kubelet/pods/03f8619e-6674-4b0d-ba30-437b90e382a8/volumes" Sep 11 14:16:06.099140 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:06.099126 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c29f9f8-86b9-4a72-9647-090933f9d110" path="/var/lib/kubelet/pods/2c29f9f8-86b9-4a72-9647-090933f9d110/volumes" Sep 11 14:16:06.151425 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:06.151392 2566 generic.go:356] "Generic (PLEG): container finished" podID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" containerID="d667580ed6841ac9b721941fd717bd8b2fd8c10b55f6e1c3b27abdb006588cde" exitCode=0 Sep 11 14:16:06.151616 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:06.151463 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" Sep 11 14:16:06.151616 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:06.151477 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" event={"ID":"970e6b7c-5827-4b2a-95cc-fe78e73b5b40","Type":"ContainerDied","Data":"d667580ed6841ac9b721941fd717bd8b2fd8c10b55f6e1c3b27abdb006588cde"} Sep 11 14:16:06.151616 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:06.151539 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84" event={"ID":"970e6b7c-5827-4b2a-95cc-fe78e73b5b40","Type":"ContainerDied","Data":"9410c88552b4d8439084abd3c07bbfd72b8b79685b13cbb602a29a6ae22a1a56"} Sep 11 14:16:06.151616 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:06.151563 2566 scope.go:117] "RemoveContainer" containerID="d667580ed6841ac9b721941fd717bd8b2fd8c10b55f6e1c3b27abdb006588cde" Sep 11 14:16:06.161347 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:06.161331 2566 scope.go:117] "RemoveContainer" containerID="507050beae6f498b39b6fa090128a96273671d1739e37543f233f42ab74095ed" Sep 11 14:16:06.172357 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:06.172335 2566 scope.go:117] "RemoveContainer" containerID="d667580ed6841ac9b721941fd717bd8b2fd8c10b55f6e1c3b27abdb006588cde" Sep 11 14:16:06.172649 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:16:06.172630 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d667580ed6841ac9b721941fd717bd8b2fd8c10b55f6e1c3b27abdb006588cde\": container with ID starting with d667580ed6841ac9b721941fd717bd8b2fd8c10b55f6e1c3b27abdb006588cde not found: ID does not exist" containerID="d667580ed6841ac9b721941fd717bd8b2fd8c10b55f6e1c3b27abdb006588cde" Sep 11 14:16:06.172731 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:06.172662 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d667580ed6841ac9b721941fd717bd8b2fd8c10b55f6e1c3b27abdb006588cde"} err="failed to get container status \"d667580ed6841ac9b721941fd717bd8b2fd8c10b55f6e1c3b27abdb006588cde\": rpc error: code = NotFound desc = could not find container \"d667580ed6841ac9b721941fd717bd8b2fd8c10b55f6e1c3b27abdb006588cde\": container with ID starting with d667580ed6841ac9b721941fd717bd8b2fd8c10b55f6e1c3b27abdb006588cde not found: ID does not exist" Sep 11 14:16:06.172731 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:06.172689 2566 scope.go:117] "RemoveContainer" containerID="507050beae6f498b39b6fa090128a96273671d1739e37543f233f42ab74095ed" Sep 11 14:16:06.172982 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:16:06.172962 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507050beae6f498b39b6fa090128a96273671d1739e37543f233f42ab74095ed\": container with ID starting with 507050beae6f498b39b6fa090128a96273671d1739e37543f233f42ab74095ed not found: ID does not exist" containerID="507050beae6f498b39b6fa090128a96273671d1739e37543f233f42ab74095ed" Sep 11 14:16:06.173044 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:06.172990 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507050beae6f498b39b6fa090128a96273671d1739e37543f233f42ab74095ed"} err="failed to get container status \"507050beae6f498b39b6fa090128a96273671d1739e37543f233f42ab74095ed\": rpc error: code = NotFound desc = could not find container \"507050beae6f498b39b6fa090128a96273671d1739e37543f233f42ab74095ed\": container with ID starting with 507050beae6f498b39b6fa090128a96273671d1739e37543f233f42ab74095ed not found: ID does not exist" Sep 11 14:16:06.195960 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:06.195931 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84"] Sep 11 14:16:06.207314 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:06.207287 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d609a19a-openshift-default-657d85d5cb-blv84"] Sep 11 14:16:08.098880 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:16:08.098843 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" path="/var/lib/kubelet/pods/970e6b7c-5827-4b2a-95cc-fe78e73b5b40/volumes" Sep 11 14:17:01.586491 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:01.586401 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz"] Sep 11 14:17:01.587043 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:01.586630 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" podUID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerName="istio-proxy" containerID="cri-o://aef628c492f57a195686e3b7713a21b5b011fbfaebeef7dfbc24744fb8678d47" gracePeriod=30 Sep 11 14:17:02.428839 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.428808 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf"] Sep 11 14:17:02.429243 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.429230 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f8619e-6674-4b0d-ba30-437b90e382a8" containerName="revoke-keys" Sep 11 14:17:02.429299 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.429245 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f8619e-6674-4b0d-ba30-437b90e382a8" containerName="revoke-keys" Sep 11 14:17:02.429299 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.429261 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" containerName="istio-proxy" Sep 11 14:17:02.429299 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.429267 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" containerName="istio-proxy" Sep 11 14:17:02.429299 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.429277 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856c4ec2-0156-4d6b-8a16-b99e081bb250" containerName="revoke-keys" Sep 11 14:17:02.429299 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.429283 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="856c4ec2-0156-4d6b-8a16-b99e081bb250" containerName="revoke-keys" Sep 11 14:17:02.429459 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.429348 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" containerName="istio-proxy" Sep 11 14:17:02.429459 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.429355 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="856c4ec2-0156-4d6b-8a16-b99e081bb250" containerName="revoke-keys" Sep 11 14:17:02.429459 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.429361 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f8619e-6674-4b0d-ba30-437b90e382a8" containerName="revoke-keys" Sep 11 14:17:02.429459 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.429369 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" containerName="istio-proxy" Sep 11 14:17:02.431781 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.431766 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.436143 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.436118 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-bef04b-openshift-default-dockercfg-79zm2\"" Sep 11 14:17:02.487306 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.487271 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf"] Sep 11 14:17:02.514730 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.514688 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-workload-certs\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.514984 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.514966 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-podinfo\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.515228 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.515202 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15fb504e-9451-46cf-a591-4ec1c0378ecc-istiod-ca-cert\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.515426 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.515410 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-token\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.515612 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.515595 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-data\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.515757 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.515741 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-credential-socket\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.515882 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.515868 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-workload-socket\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.516016 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.516002 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-envoy\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.516162 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.516146 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkjd2\" (UniqueName: \"kubernetes.io/projected/15fb504e-9451-46cf-a591-4ec1c0378ecc-kube-api-access-tkjd2\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.616750 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.616711 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-credential-socket\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.617216 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.616758 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-workload-socket\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.617216 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.616788 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-envoy\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.617216 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.616807 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkjd2\" (UniqueName: \"kubernetes.io/projected/15fb504e-9451-46cf-a591-4ec1c0378ecc-kube-api-access-tkjd2\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.617216 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.616829 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-workload-certs\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.617216 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.616842 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-podinfo\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.617216 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.616909 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15fb504e-9451-46cf-a591-4ec1c0378ecc-istiod-ca-cert\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.617216 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.616967 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-token\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.617216 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.617000 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-data\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.617216 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.617191 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-credential-socket\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.617642 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.617618 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-workload-certs\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.617710 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.617689 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15fb504e-9451-46cf-a591-4ec1c0378ecc-istiod-ca-cert\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.617766 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.617729 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-workload-socket\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.617808 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.617735 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-data\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.619556 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.619526 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-podinfo\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.619660 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.619627 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-envoy\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.631575 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.631555 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-token\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.635482 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.635463 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkjd2\" (UniqueName: \"kubernetes.io/projected/15fb504e-9451-46cf-a591-4ec1c0378ecc-kube-api-access-tkjd2\") pod \"e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:02.741812 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:02.741776 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:03.040234 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:03.040207 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf"] Sep 11 14:17:03.040675 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:17:03.040652 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fb504e_9451_46cf_a591_4ec1c0378ecc.slice/crio-f53802cffbe13972d7f444a256ada94016a677c16ab182bd6a1932534f2e3fd8 WatchSource:0}: Error finding container f53802cffbe13972d7f444a256ada94016a677c16ab182bd6a1932534f2e3fd8: Status 404 returned error can't find the container with id f53802cffbe13972d7f444a256ada94016a677c16ab182bd6a1932534f2e3fd8 Sep 11 14:17:03.042863 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:03.042829 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:17:03.042925 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:03.042897 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:17:03.042966 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:03.042929 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:17:03.532174 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:03.532135 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" event={"ID":"15fb504e-9451-46cf-a591-4ec1c0378ecc","Type":"ContainerStarted","Data":"bd6d1cb35e0288aa4f2100e2d882a56bebac1e632f1254de9390465dbfd573fd"} Sep 11 14:17:03.532364 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:03.532205 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" event={"ID":"15fb504e-9451-46cf-a591-4ec1c0378ecc","Type":"ContainerStarted","Data":"f53802cffbe13972d7f444a256ada94016a677c16ab182bd6a1932534f2e3fd8"} Sep 11 14:17:03.742354 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:03.742317 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:04.747580 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:04.747549 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:04.785785 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:04.785737 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" podStartSLOduration=2.785722136 podStartE2EDuration="2.785722136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:03.611180039 +0000 UTC m=+1164.016161167" watchObservedRunningTime="2026-09-11 14:17:04.785722136 +0000 UTC m=+1165.190703263" Sep 11 14:17:05.545652 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:05.545620 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:05.546797 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:05.546774 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:06.342142 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.342106 2566 patch_prober.go:28] interesting pod/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:17:06.342519 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.342178 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" podUID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:17:06.835104 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.835069 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:17:06.957925 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.957847 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvvsv\" (UniqueName: \"kubernetes.io/projected/ad25e1d0-b107-4042-86d0-9026e5c78517-kube-api-access-nvvsv\") pod \"ad25e1d0-b107-4042-86d0-9026e5c78517\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " Sep 11 14:17:06.957925 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.957877 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-token\") pod \"ad25e1d0-b107-4042-86d0-9026e5c78517\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " Sep 11 14:17:06.957925 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.957896 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-workload-socket\") pod \"ad25e1d0-b107-4042-86d0-9026e5c78517\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " Sep 11 14:17:06.958264 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.957953 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad25e1d0-b107-4042-86d0-9026e5c78517-istiod-ca-cert\") pod \"ad25e1d0-b107-4042-86d0-9026e5c78517\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " Sep 11 14:17:06.958264 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.957977 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-podinfo\") pod \"ad25e1d0-b107-4042-86d0-9026e5c78517\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " Sep 11 14:17:06.958264 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.958019 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-credential-socket\") pod \"ad25e1d0-b107-4042-86d0-9026e5c78517\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " Sep 11 14:17:06.958264 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.958060 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-envoy\") pod \"ad25e1d0-b107-4042-86d0-9026e5c78517\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " Sep 11 14:17:06.958264 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.958187 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-data\") pod \"ad25e1d0-b107-4042-86d0-9026e5c78517\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " Sep 11 14:17:06.958264 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.958223 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-workload-certs\") pod \"ad25e1d0-b107-4042-86d0-9026e5c78517\" (UID: \"ad25e1d0-b107-4042-86d0-9026e5c78517\") " Sep 11 14:17:06.958588 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.958292 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ad25e1d0-b107-4042-86d0-9026e5c78517" (UID: "ad25e1d0-b107-4042-86d0-9026e5c78517"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:17:06.958588 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.958394 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad25e1d0-b107-4042-86d0-9026e5c78517-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ad25e1d0-b107-4042-86d0-9026e5c78517" (UID: "ad25e1d0-b107-4042-86d0-9026e5c78517"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:17:06.958588 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.958449 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ad25e1d0-b107-4042-86d0-9026e5c78517" (UID: "ad25e1d0-b107-4042-86d0-9026e5c78517"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:17:06.958588 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.958536 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-data" (OuterVolumeSpecName: "istio-data") pod "ad25e1d0-b107-4042-86d0-9026e5c78517" (UID: "ad25e1d0-b107-4042-86d0-9026e5c78517"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:17:06.958588 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.958567 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ad25e1d0-b107-4042-86d0-9026e5c78517" (UID: "ad25e1d0-b107-4042-86d0-9026e5c78517"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:17:06.959022 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.958670 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:17:06.959022 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.958688 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:17:06.959022 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.958703 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:17:06.959022 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.958714 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad25e1d0-b107-4042-86d0-9026e5c78517-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:17:06.959022 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.958729 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:17:06.960372 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.960341 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-token" (OuterVolumeSpecName: "istio-token") pod "ad25e1d0-b107-4042-86d0-9026e5c78517" (UID: "ad25e1d0-b107-4042-86d0-9026e5c78517"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:17:06.960372 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.960347 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad25e1d0-b107-4042-86d0-9026e5c78517-kube-api-access-nvvsv" (OuterVolumeSpecName: "kube-api-access-nvvsv") pod "ad25e1d0-b107-4042-86d0-9026e5c78517" (UID: "ad25e1d0-b107-4042-86d0-9026e5c78517"). InnerVolumeSpecName "kube-api-access-nvvsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:17:06.960536 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.960405 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ad25e1d0-b107-4042-86d0-9026e5c78517" (UID: "ad25e1d0-b107-4042-86d0-9026e5c78517"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:17:06.960536 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:06.960436 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ad25e1d0-b107-4042-86d0-9026e5c78517" (UID: "ad25e1d0-b107-4042-86d0-9026e5c78517"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:17:07.059417 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.059386 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvvsv\" (UniqueName: \"kubernetes.io/projected/ad25e1d0-b107-4042-86d0-9026e5c78517-kube-api-access-nvvsv\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:17:07.059417 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.059410 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:17:07.059417 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.059420 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:17:07.059646 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.059428 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad25e1d0-b107-4042-86d0-9026e5c78517-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:17:07.558930 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.558897 2566 generic.go:356] "Generic (PLEG): container finished" podID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerID="aef628c492f57a195686e3b7713a21b5b011fbfaebeef7dfbc24744fb8678d47" exitCode=0 Sep 11 14:17:07.559453 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.558956 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" Sep 11 14:17:07.559453 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.558997 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" event={"ID":"ad25e1d0-b107-4042-86d0-9026e5c78517","Type":"ContainerDied","Data":"aef628c492f57a195686e3b7713a21b5b011fbfaebeef7dfbc24744fb8678d47"} Sep 11 14:17:07.559453 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.559070 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz" event={"ID":"ad25e1d0-b107-4042-86d0-9026e5c78517","Type":"ContainerDied","Data":"7ad7386056bf73f91346f4a7ce72d86a781ebb3dd096a1ecb799e5db30fbd7d0"} Sep 11 14:17:07.559453 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.559122 2566 scope.go:117] "RemoveContainer" containerID="aef628c492f57a195686e3b7713a21b5b011fbfaebeef7dfbc24744fb8678d47" Sep 11 14:17:07.571029 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.571006 2566 scope.go:117] "RemoveContainer" containerID="8eda68b74570ac1e091bac83252dfc2e584ce66c1fab5746cc80cecddca8bba6" Sep 11 14:17:07.580283 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.580262 2566 scope.go:117] "RemoveContainer" containerID="aef628c492f57a195686e3b7713a21b5b011fbfaebeef7dfbc24744fb8678d47" Sep 11 14:17:07.580571 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:17:07.580553 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef628c492f57a195686e3b7713a21b5b011fbfaebeef7dfbc24744fb8678d47\": container with ID starting with aef628c492f57a195686e3b7713a21b5b011fbfaebeef7dfbc24744fb8678d47 not found: ID does not exist" containerID="aef628c492f57a195686e3b7713a21b5b011fbfaebeef7dfbc24744fb8678d47" Sep 11 14:17:07.580619 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.580580 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef628c492f57a195686e3b7713a21b5b011fbfaebeef7dfbc24744fb8678d47"} err="failed to get container status \"aef628c492f57a195686e3b7713a21b5b011fbfaebeef7dfbc24744fb8678d47\": rpc error: code = NotFound desc = could not find container \"aef628c492f57a195686e3b7713a21b5b011fbfaebeef7dfbc24744fb8678d47\": container with ID starting with aef628c492f57a195686e3b7713a21b5b011fbfaebeef7dfbc24744fb8678d47 not found: ID does not exist" Sep 11 14:17:07.580619 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.580599 2566 scope.go:117] "RemoveContainer" containerID="8eda68b74570ac1e091bac83252dfc2e584ce66c1fab5746cc80cecddca8bba6" Sep 11 14:17:07.580943 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:17:07.580919 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eda68b74570ac1e091bac83252dfc2e584ce66c1fab5746cc80cecddca8bba6\": container with ID starting with 8eda68b74570ac1e091bac83252dfc2e584ce66c1fab5746cc80cecddca8bba6 not found: ID does not exist" containerID="8eda68b74570ac1e091bac83252dfc2e584ce66c1fab5746cc80cecddca8bba6" Sep 11 14:17:07.581017 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.580951 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eda68b74570ac1e091bac83252dfc2e584ce66c1fab5746cc80cecddca8bba6"} err="failed to get container status \"8eda68b74570ac1e091bac83252dfc2e584ce66c1fab5746cc80cecddca8bba6\": rpc error: code = NotFound desc = could not find container \"8eda68b74570ac1e091bac83252dfc2e584ce66c1fab5746cc80cecddca8bba6\": container with ID starting with 8eda68b74570ac1e091bac83252dfc2e584ce66c1fab5746cc80cecddca8bba6 not found: ID does not exist" Sep 11 14:17:07.602720 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.602689 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz"] Sep 11 14:17:07.619446 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:07.619419 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-2a21d9-openshift-default-fdd64446d-p6wsz"] Sep 11 14:17:08.099277 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:08.099243 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad25e1d0-b107-4042-86d0-9026e5c78517" path="/var/lib/kubelet/pods/ad25e1d0-b107-4042-86d0-9026e5c78517/volumes" Sep 11 14:17:26.690165 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:26.690131 2566 generic.go:356] "Generic (PLEG): container finished" podID="15fb504e-9451-46cf-a591-4ec1c0378ecc" containerID="bd6d1cb35e0288aa4f2100e2d882a56bebac1e632f1254de9390465dbfd573fd" exitCode=0 Sep 11 14:17:26.690612 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:26.690206 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" event={"ID":"15fb504e-9451-46cf-a591-4ec1c0378ecc","Type":"ContainerDied","Data":"bd6d1cb35e0288aa4f2100e2d882a56bebac1e632f1254de9390465dbfd573fd"} Sep 11 14:17:26.690670 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:26.690628 2566 scope.go:117] "RemoveContainer" containerID="bd6d1cb35e0288aa4f2100e2d882a56bebac1e632f1254de9390465dbfd573fd" Sep 11 14:17:26.691663 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:26.691641 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:17:26.691721 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:26.691706 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:17:26.691755 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:26.691738 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:17:27.699294 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:27.699256 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" event={"ID":"15fb504e-9451-46cf-a591-4ec1c0378ecc","Type":"ContainerStarted","Data":"c39d9e58c3add6d61bb1ccc2d82e4396ec7a379af598ab49ddfd6b0c9e2b3b6c"} Sep 11 14:17:27.742640 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:27.742604 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:27.743009 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:27.742925 2566 patch_prober.go:28] interesting pod/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 11 14:17:27.743209 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:27.743174 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" podUID="15fb504e-9451-46cf-a591-4ec1c0378ecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 11 14:17:28.742900 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:28.742855 2566 patch_prober.go:28] interesting pod/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 11 14:17:28.743351 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:28.742923 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" podUID="15fb504e-9451-46cf-a591-4ec1c0378ecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 11 14:17:29.742959 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:29.742919 2566 patch_prober.go:28] interesting pod/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 11 14:17:29.743389 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:29.742975 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" podUID="15fb504e-9451-46cf-a591-4ec1c0378ecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 11 14:17:30.743063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:30.743020 2566 patch_prober.go:28] interesting pod/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 11 14:17:30.743483 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:30.743106 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" podUID="15fb504e-9451-46cf-a591-4ec1c0378ecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 11 14:17:31.742939 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:31.742906 2566 patch_prober.go:28] interesting pod/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 11 14:17:31.743152 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:31.742954 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" podUID="15fb504e-9451-46cf-a591-4ec1c0378ecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 11 14:17:32.742413 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:32.742381 2566 patch_prober.go:28] interesting pod/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 11 14:17:32.742676 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:32.742437 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" podUID="15fb504e-9451-46cf-a591-4ec1c0378ecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 11 14:17:33.747459 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:33.747427 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:33.747973 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:33.747719 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:33.748509 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:33.748492 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:17:40.153360 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:40.153329 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:17:40.154016 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:17:40.153995 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:18:02.355462 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.355428 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277"] Sep 11 14:18:02.356045 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.356025 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerName="istio-proxy" Sep 11 14:18:02.356164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.356049 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerName="istio-proxy" Sep 11 14:18:02.356164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.356100 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerName="istio-proxy" Sep 11 14:18:02.356164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.356110 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerName="istio-proxy" Sep 11 14:18:02.356164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.356120 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" containerName="istio-proxy" Sep 11 14:18:02.356164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.356131 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="970e6b7c-5827-4b2a-95cc-fe78e73b5b40" containerName="istio-proxy" Sep 11 14:18:02.356414 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.356256 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerName="istio-proxy" Sep 11 14:18:02.356414 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.356269 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad25e1d0-b107-4042-86d0-9026e5c78517" containerName="istio-proxy" Sep 11 14:18:02.359195 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.359173 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.362755 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.362737 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-20588104-openshift-default-dockercfg-4d8df\"" Sep 11 14:18:02.395354 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.395325 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277"] Sep 11 14:18:02.400821 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.400368 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/410f7071-8ff3-44d6-84a7-921ad9624ff9-istiod-ca-cert\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.400821 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.400414 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px9g4\" (UniqueName: \"kubernetes.io/projected/410f7071-8ff3-44d6-84a7-921ad9624ff9-kube-api-access-px9g4\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.400821 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.400452 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-token\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.400821 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.400487 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-workload-certs\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.400821 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.400515 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-podinfo\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.400821 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.400588 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-credential-socket\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.400821 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.400634 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-workload-socket\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.400821 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.400668 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-envoy\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.400821 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.400701 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-data\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.501230 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.501185 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-workload-socket\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.501230 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.501231 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-envoy\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.501485 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.501374 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-data\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.501485 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.501471 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/410f7071-8ff3-44d6-84a7-921ad9624ff9-istiod-ca-cert\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.501584 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.501503 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9g4\" (UniqueName: \"kubernetes.io/projected/410f7071-8ff3-44d6-84a7-921ad9624ff9-kube-api-access-px9g4\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.501584 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.501541 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-token\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.501584 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.501579 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-workload-certs\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.501760 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.501609 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-podinfo\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.501760 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.501692 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-credential-socket\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.501760 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.501735 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-data\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.502230 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.502207 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-workload-certs\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.502413 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.502386 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-credential-socket\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.502516 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.502411 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/410f7071-8ff3-44d6-84a7-921ad9624ff9-istiod-ca-cert\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.502516 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.502275 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-workload-socket\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.503853 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.503823 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-envoy\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.504587 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.504569 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-podinfo\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.536517 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.536485 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9g4\" (UniqueName: \"kubernetes.io/projected/410f7071-8ff3-44d6-84a7-921ad9624ff9-kube-api-access-px9g4\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.537459 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.537401 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-token\") pod \"e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.669028 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.668940 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:02.963017 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.962991 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277"] Sep 11 14:18:02.964347 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:18:02.964317 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod410f7071_8ff3_44d6_84a7_921ad9624ff9.slice/crio-793a6048a1e34a49f3f71451ae42df92b8573b4a4082282d11889b27398d8d43 WatchSource:0}: Error finding container 793a6048a1e34a49f3f71451ae42df92b8573b4a4082282d11889b27398d8d43: Status 404 returned error can't find the container with id 793a6048a1e34a49f3f71451ae42df92b8573b4a4082282d11889b27398d8d43 Sep 11 14:18:02.968136 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.967010 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:02.968136 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.967115 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:02.968136 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:02.967176 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:03.949412 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:03.949377 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" event={"ID":"410f7071-8ff3-44d6-84a7-921ad9624ff9","Type":"ContainerStarted","Data":"e7c8a547b649cab07930674c89047387ce7b0420be9f0090a9bf658c13a9e3e0"} Sep 11 14:18:03.949821 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:03.949426 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" event={"ID":"410f7071-8ff3-44d6-84a7-921ad9624ff9","Type":"ContainerStarted","Data":"793a6048a1e34a49f3f71451ae42df92b8573b4a4082282d11889b27398d8d43"} Sep 11 14:18:04.669613 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:04.669576 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:04.675230 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:04.675205 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:04.713805 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:04.713727 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" podStartSLOduration=2.71370931 podStartE2EDuration="2.71370931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:04.011695421 +0000 UTC m=+1224.416676561" watchObservedRunningTime="2026-09-11 14:18:04.71370931 +0000 UTC m=+1225.118690437" Sep 11 14:18:04.960488 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:04.960403 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:04.964103 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:04.964061 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:15.218586 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:15.218548 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bef04b-d9b95"] Sep 11 14:18:15.222150 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:15.222134 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bef04b-d9b95" Sep 11 14:18:15.225835 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:15.225816 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:18:15.237110 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:15.237068 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bef04b-d9b95"] Sep 11 14:18:15.327772 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:15.327733 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57626d5b-2348-4255-b36d-9461ab0c7587-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-bef04b-d9b95\" (UID: \"57626d5b-2348-4255-b36d-9461ab0c7587\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bef04b-d9b95" Sep 11 14:18:15.428957 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:15.428920 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57626d5b-2348-4255-b36d-9461ab0c7587-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-bef04b-d9b95\" (UID: \"57626d5b-2348-4255-b36d-9461ab0c7587\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bef04b-d9b95" Sep 11 14:18:15.429550 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:15.429528 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57626d5b-2348-4255-b36d-9461ab0c7587-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-bef04b-d9b95\" (UID: \"57626d5b-2348-4255-b36d-9461ab0c7587\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bef04b-d9b95" Sep 11 14:18:15.532936 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:15.532891 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bef04b-d9b95" Sep 11 14:18:15.816344 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:15.816320 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bef04b-d9b95"] Sep 11 14:18:15.818263 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:18:15.818236 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57626d5b_2348_4255_b36d_9461ab0c7587.slice/crio-c92354dccfed4920910df53f3cf92968cee5849a9af2689379aec9dc6ae210a1 WatchSource:0}: Error finding container c92354dccfed4920910df53f3cf92968cee5849a9af2689379aec9dc6ae210a1: Status 404 returned error can't find the container with id c92354dccfed4920910df53f3cf92968cee5849a9af2689379aec9dc6ae210a1 Sep 11 14:18:16.045893 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:16.045855 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bef04b-d9b95" event={"ID":"57626d5b-2348-4255-b36d-9461ab0c7587","Type":"ContainerStarted","Data":"c92354dccfed4920910df53f3cf92968cee5849a9af2689379aec9dc6ae210a1"} Sep 11 14:18:17.053414 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:17.053377 2566 generic.go:356] "Generic (PLEG): container finished" podID="57626d5b-2348-4255-b36d-9461ab0c7587" containerID="6b5a9a1e7a634c90b1b00de50a41cbc5719d4f5fd8320a9790f418a52c92a90c" exitCode=0 Sep 11 14:18:17.053791 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:17.053463 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bef04b-d9b95" event={"ID":"57626d5b-2348-4255-b36d-9461ab0c7587","Type":"ContainerDied","Data":"6b5a9a1e7a634c90b1b00de50a41cbc5719d4f5fd8320a9790f418a52c92a90c"} Sep 11 14:18:18.203765 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:18.203740 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bef04b-d9b95" Sep 11 14:18:18.361929 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:18.361838 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57626d5b-2348-4255-b36d-9461ab0c7587-maas-api-service-ca\") pod \"57626d5b-2348-4255-b36d-9461ab0c7587\" (UID: \"57626d5b-2348-4255-b36d-9461ab0c7587\") " Sep 11 14:18:18.362307 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:18.362275 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57626d5b-2348-4255-b36d-9461ab0c7587-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "57626d5b-2348-4255-b36d-9461ab0c7587" (UID: "57626d5b-2348-4255-b36d-9461ab0c7587"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:18.463114 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:18.463060 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57626d5b-2348-4255-b36d-9461ab0c7587-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:18:19.067315 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:19.067285 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bef04b-d9b95" Sep 11 14:18:19.067499 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:19.067328 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bef04b-d9b95" event={"ID":"57626d5b-2348-4255-b36d-9461ab0c7587","Type":"ContainerDied","Data":"c92354dccfed4920910df53f3cf92968cee5849a9af2689379aec9dc6ae210a1"} Sep 11 14:18:19.067499 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:19.067378 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c92354dccfed4920910df53f3cf92968cee5849a9af2689379aec9dc6ae210a1" Sep 11 14:18:26.636306 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:26.636270 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bef04b-d9b95"] Sep 11 14:18:26.651182 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:26.651145 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-bef04b-d9b95"] Sep 11 14:18:28.100491 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:28.100458 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57626d5b-2348-4255-b36d-9461ab0c7587" path="/var/lib/kubelet/pods/57626d5b-2348-4255-b36d-9461ab0c7587/volumes" Sep 11 14:18:29.148860 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:29.148779 2566 generic.go:356] "Generic (PLEG): container finished" podID="410f7071-8ff3-44d6-84a7-921ad9624ff9" containerID="e7c8a547b649cab07930674c89047387ce7b0420be9f0090a9bf658c13a9e3e0" exitCode=0 Sep 11 14:18:29.148860 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:29.148833 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" event={"ID":"410f7071-8ff3-44d6-84a7-921ad9624ff9","Type":"ContainerDied","Data":"e7c8a547b649cab07930674c89047387ce7b0420be9f0090a9bf658c13a9e3e0"} Sep 11 14:18:29.149329 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:29.149251 2566 scope.go:117] "RemoveContainer" containerID="e7c8a547b649cab07930674c89047387ce7b0420be9f0090a9bf658c13a9e3e0" Sep 11 14:18:29.150429 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:29.150402 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:29.150494 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:29.150475 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:29.150536 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:29.150506 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:30.157457 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:30.157422 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" event={"ID":"410f7071-8ff3-44d6-84a7-921ad9624ff9","Type":"ContainerStarted","Data":"e0426f1639d4f78ba9dd5d6fdebc2e2ae8ebce06a8634a707fe1693799a21212"} Sep 11 14:18:30.669734 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:30.669699 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:30.671043 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:30.671014 2566 patch_prober.go:28] interesting pod/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 11 14:18:30.671183 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:30.671105 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" podUID="410f7071-8ff3-44d6-84a7-921ad9624ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 11 14:18:31.669530 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:31.669454 2566 patch_prober.go:28] interesting pod/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 11 14:18:31.669530 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:31.669508 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" podUID="410f7071-8ff3-44d6-84a7-921ad9624ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 11 14:18:32.669601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:32.669567 2566 patch_prober.go:28] interesting pod/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 11 14:18:32.670029 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:32.669623 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" podUID="410f7071-8ff3-44d6-84a7-921ad9624ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 11 14:18:33.670137 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:33.670066 2566 patch_prober.go:28] interesting pod/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 11 14:18:33.670529 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:33.670158 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" podUID="410f7071-8ff3-44d6-84a7-921ad9624ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 11 14:18:34.669935 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:34.669899 2566 patch_prober.go:28] interesting pod/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 11 14:18:34.670218 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:34.669966 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" podUID="410f7071-8ff3-44d6-84a7-921ad9624ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 11 14:18:34.960518 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:34.960430 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:35.528962 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:35.528929 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf"] Sep 11 14:18:35.529175 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:35.529154 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" podUID="15fb504e-9451-46cf-a591-4ec1c0378ecc" containerName="istio-proxy" containerID="cri-o://c39d9e58c3add6d61bb1ccc2d82e4396ec7a379af598ab49ddfd6b0c9e2b3b6c" gracePeriod=30 Sep 11 14:18:35.674063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:35.674035 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:35.675124 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:35.675104 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:18:36.375862 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.375829 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k"] Sep 11 14:18:36.376328 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.376302 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57626d5b-2348-4255-b36d-9461ab0c7587" containerName="revoke-keys" Sep 11 14:18:36.376328 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.376325 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="57626d5b-2348-4255-b36d-9461ab0c7587" containerName="revoke-keys" Sep 11 14:18:36.376519 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.376443 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="57626d5b-2348-4255-b36d-9461ab0c7587" containerName="revoke-keys" Sep 11 14:18:36.378912 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.378894 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.381864 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.381845 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-ea85ff-openshift-default-dockercfg-rp9gd\"" Sep 11 14:18:36.408580 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.408545 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k"] Sep 11 14:18:36.460626 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.460593 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-workload-certs\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.460796 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.460633 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-envoy\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.460796 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.460662 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-data\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.460796 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.460744 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7s9r\" (UniqueName: \"kubernetes.io/projected/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-kube-api-access-k7s9r\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.460796 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.460778 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-workload-socket\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.460983 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.460796 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-credential-socket\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.460983 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.460878 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istiod-ca-cert\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.460983 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.460918 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-token\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.460983 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.460943 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-podinfo\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.562049 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.562017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7s9r\" (UniqueName: \"kubernetes.io/projected/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-kube-api-access-k7s9r\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.562259 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.562056 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-workload-socket\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.562259 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.562074 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-credential-socket\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.562259 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.562133 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istiod-ca-cert\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.562259 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.562163 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-token\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.562259 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.562206 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-podinfo\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.562540 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.562286 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-workload-certs\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.562540 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.562329 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-envoy\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.562540 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.562369 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-data\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.562540 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.562493 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-workload-socket\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.562748 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.562553 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-credential-socket\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.562748 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.562723 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-data\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.563454 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.563347 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-workload-certs\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.563454 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.563375 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istiod-ca-cert\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.566737 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.566573 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-podinfo\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.567329 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.567279 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-envoy\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.579841 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.579789 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7s9r\" (UniqueName: \"kubernetes.io/projected/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-kube-api-access-k7s9r\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.584115 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.583825 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-token\") pod \"e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.689207 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.689114 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:36.987869 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.987839 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k"] Sep 11 14:18:36.988616 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:18:36.988583 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc622c3b1_cddb_47e4_97e2_d5d67f858a2f.slice/crio-e54e2e83d3e460579c549f464f08ce362b0948c0666d7f5c2bbf0f051a8e841a WatchSource:0}: Error finding container e54e2e83d3e460579c549f464f08ce362b0948c0666d7f5c2bbf0f051a8e841a: Status 404 returned error can't find the container with id e54e2e83d3e460579c549f464f08ce362b0948c0666d7f5c2bbf0f051a8e841a Sep 11 14:18:36.990963 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.990922 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:36.991053 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.990998 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:36.991053 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:36.991028 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:37.207522 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:37.207470 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" event={"ID":"c622c3b1-cddb-47e4-97e2-d5d67f858a2f","Type":"ContainerStarted","Data":"6bcf987564484e3212e35b988f1554b1701008cb2cc9316fca86ea7e140f742f"} Sep 11 14:18:37.207708 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:37.207533 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" event={"ID":"c622c3b1-cddb-47e4-97e2-d5d67f858a2f","Type":"ContainerStarted","Data":"e54e2e83d3e460579c549f464f08ce362b0948c0666d7f5c2bbf0f051a8e841a"} Sep 11 14:18:37.238679 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:37.238573 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" podStartSLOduration=1.238554807 podStartE2EDuration="1.238554807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:37.234713915 +0000 UTC m=+1257.639695042" watchObservedRunningTime="2026-09-11 14:18:37.238554807 +0000 UTC m=+1257.643535936" Sep 11 14:18:37.690334 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:37.690246 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:38.695543 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:38.695505 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:39.220766 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:39.220731 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:39.221887 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:39.221869 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:18:40.784692 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.784655 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:18:40.906351 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.906255 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-workload-socket\") pod \"15fb504e-9451-46cf-a591-4ec1c0378ecc\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " Sep 11 14:18:40.906351 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.906301 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-workload-certs\") pod \"15fb504e-9451-46cf-a591-4ec1c0378ecc\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " Sep 11 14:18:40.906351 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.906324 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15fb504e-9451-46cf-a591-4ec1c0378ecc-istiod-ca-cert\") pod \"15fb504e-9451-46cf-a591-4ec1c0378ecc\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " Sep 11 14:18:40.906351 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.906352 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-token\") pod \"15fb504e-9451-46cf-a591-4ec1c0378ecc\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " Sep 11 14:18:40.906699 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.906371 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-envoy\") pod \"15fb504e-9451-46cf-a591-4ec1c0378ecc\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " Sep 11 14:18:40.906699 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.906403 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-credential-socket\") pod \"15fb504e-9451-46cf-a591-4ec1c0378ecc\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " Sep 11 14:18:40.906699 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.906437 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkjd2\" (UniqueName: \"kubernetes.io/projected/15fb504e-9451-46cf-a591-4ec1c0378ecc-kube-api-access-tkjd2\") pod \"15fb504e-9451-46cf-a591-4ec1c0378ecc\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " Sep 11 14:18:40.906699 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.906507 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-data\") pod \"15fb504e-9451-46cf-a591-4ec1c0378ecc\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " Sep 11 14:18:40.906699 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.906553 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-podinfo\") pod \"15fb504e-9451-46cf-a591-4ec1c0378ecc\" (UID: \"15fb504e-9451-46cf-a591-4ec1c0378ecc\") " Sep 11 14:18:40.906699 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.906589 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "15fb504e-9451-46cf-a591-4ec1c0378ecc" (UID: "15fb504e-9451-46cf-a591-4ec1c0378ecc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:18:40.906699 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.906615 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "15fb504e-9451-46cf-a591-4ec1c0378ecc" (UID: "15fb504e-9451-46cf-a591-4ec1c0378ecc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:18:40.907042 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.906790 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-data" (OuterVolumeSpecName: "istio-data") pod "15fb504e-9451-46cf-a591-4ec1c0378ecc" (UID: "15fb504e-9451-46cf-a591-4ec1c0378ecc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:18:40.907042 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.906867 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15fb504e-9451-46cf-a591-4ec1c0378ecc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "15fb504e-9451-46cf-a591-4ec1c0378ecc" (UID: "15fb504e-9451-46cf-a591-4ec1c0378ecc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:40.907042 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.906972 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "15fb504e-9451-46cf-a591-4ec1c0378ecc" (UID: "15fb504e-9451-46cf-a591-4ec1c0378ecc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:18:40.907235 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.907121 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:18:40.907235 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.907139 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:18:40.907235 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.907153 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15fb504e-9451-46cf-a591-4ec1c0378ecc-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:18:40.907235 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.907167 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:18:40.907235 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.907180 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:18:40.908946 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.908919 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fb504e-9451-46cf-a591-4ec1c0378ecc-kube-api-access-tkjd2" (OuterVolumeSpecName: "kube-api-access-tkjd2") pod "15fb504e-9451-46cf-a591-4ec1c0378ecc" (UID: "15fb504e-9451-46cf-a591-4ec1c0378ecc"). InnerVolumeSpecName "kube-api-access-tkjd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:18:40.909058 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.908944 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "15fb504e-9451-46cf-a591-4ec1c0378ecc" (UID: "15fb504e-9451-46cf-a591-4ec1c0378ecc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:18:40.909058 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.908951 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-token" (OuterVolumeSpecName: "istio-token") pod "15fb504e-9451-46cf-a591-4ec1c0378ecc" (UID: "15fb504e-9451-46cf-a591-4ec1c0378ecc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:18:40.909241 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:40.909227 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "15fb504e-9451-46cf-a591-4ec1c0378ecc" (UID: "15fb504e-9451-46cf-a591-4ec1c0378ecc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:18:41.008453 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.008419 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:18:41.008453 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.008449 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:18:41.008453 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.008461 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkjd2\" (UniqueName: \"kubernetes.io/projected/15fb504e-9451-46cf-a591-4ec1c0378ecc-kube-api-access-tkjd2\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:18:41.008691 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.008472 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15fb504e-9451-46cf-a591-4ec1c0378ecc-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:18:41.242165 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.242129 2566 generic.go:356] "Generic (PLEG): container finished" podID="15fb504e-9451-46cf-a591-4ec1c0378ecc" containerID="c39d9e58c3add6d61bb1ccc2d82e4396ec7a379af598ab49ddfd6b0c9e2b3b6c" exitCode=0 Sep 11 14:18:41.242340 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.242207 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" Sep 11 14:18:41.242340 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.242217 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" event={"ID":"15fb504e-9451-46cf-a591-4ec1c0378ecc","Type":"ContainerDied","Data":"c39d9e58c3add6d61bb1ccc2d82e4396ec7a379af598ab49ddfd6b0c9e2b3b6c"} Sep 11 14:18:41.242340 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.242292 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf" event={"ID":"15fb504e-9451-46cf-a591-4ec1c0378ecc","Type":"ContainerDied","Data":"f53802cffbe13972d7f444a256ada94016a677c16ab182bd6a1932534f2e3fd8"} Sep 11 14:18:41.242340 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.242314 2566 scope.go:117] "RemoveContainer" containerID="c39d9e58c3add6d61bb1ccc2d82e4396ec7a379af598ab49ddfd6b0c9e2b3b6c" Sep 11 14:18:41.252808 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.252793 2566 scope.go:117] "RemoveContainer" containerID="bd6d1cb35e0288aa4f2100e2d882a56bebac1e632f1254de9390465dbfd573fd" Sep 11 14:18:41.261716 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.261697 2566 scope.go:117] "RemoveContainer" containerID="c39d9e58c3add6d61bb1ccc2d82e4396ec7a379af598ab49ddfd6b0c9e2b3b6c" Sep 11 14:18:41.261975 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:18:41.261958 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39d9e58c3add6d61bb1ccc2d82e4396ec7a379af598ab49ddfd6b0c9e2b3b6c\": container with ID starting with c39d9e58c3add6d61bb1ccc2d82e4396ec7a379af598ab49ddfd6b0c9e2b3b6c not found: ID does not exist" containerID="c39d9e58c3add6d61bb1ccc2d82e4396ec7a379af598ab49ddfd6b0c9e2b3b6c" Sep 11 14:18:41.262025 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.261985 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39d9e58c3add6d61bb1ccc2d82e4396ec7a379af598ab49ddfd6b0c9e2b3b6c"} err="failed to get container status \"c39d9e58c3add6d61bb1ccc2d82e4396ec7a379af598ab49ddfd6b0c9e2b3b6c\": rpc error: code = NotFound desc = could not find container \"c39d9e58c3add6d61bb1ccc2d82e4396ec7a379af598ab49ddfd6b0c9e2b3b6c\": container with ID starting with c39d9e58c3add6d61bb1ccc2d82e4396ec7a379af598ab49ddfd6b0c9e2b3b6c not found: ID does not exist" Sep 11 14:18:41.262025 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.262002 2566 scope.go:117] "RemoveContainer" containerID="bd6d1cb35e0288aa4f2100e2d882a56bebac1e632f1254de9390465dbfd573fd" Sep 11 14:18:41.262285 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:18:41.262269 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6d1cb35e0288aa4f2100e2d882a56bebac1e632f1254de9390465dbfd573fd\": container with ID starting with bd6d1cb35e0288aa4f2100e2d882a56bebac1e632f1254de9390465dbfd573fd not found: ID does not exist" containerID="bd6d1cb35e0288aa4f2100e2d882a56bebac1e632f1254de9390465dbfd573fd" Sep 11 14:18:41.262349 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.262289 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6d1cb35e0288aa4f2100e2d882a56bebac1e632f1254de9390465dbfd573fd"} err="failed to get container status \"bd6d1cb35e0288aa4f2100e2d882a56bebac1e632f1254de9390465dbfd573fd\": rpc error: code = NotFound desc = could not find container \"bd6d1cb35e0288aa4f2100e2d882a56bebac1e632f1254de9390465dbfd573fd\": container with ID starting with bd6d1cb35e0288aa4f2100e2d882a56bebac1e632f1254de9390465dbfd573fd not found: ID does not exist" Sep 11 14:18:41.269601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.269582 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf"] Sep 11 14:18:41.283691 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:41.283668 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-bef04b-openshift-default-777d6c5548-dfklf"] Sep 11 14:18:42.097726 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:42.097689 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15fb504e-9451-46cf-a591-4ec1c0378ecc" path="/var/lib/kubelet/pods/15fb504e-9451-46cf-a591-4ec1c0378ecc/volumes" Sep 11 14:18:46.381701 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:46.381667 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k"] Sep 11 14:18:46.382134 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:46.382120 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15fb504e-9451-46cf-a591-4ec1c0378ecc" containerName="istio-proxy" Sep 11 14:18:46.382134 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:46.382135 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15fb504e-9451-46cf-a591-4ec1c0378ecc" containerName="istio-proxy" Sep 11 14:18:46.382222 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:46.382146 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15fb504e-9451-46cf-a591-4ec1c0378ecc" containerName="istio-proxy" Sep 11 14:18:46.382222 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:46.382152 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="15fb504e-9451-46cf-a591-4ec1c0378ecc" containerName="istio-proxy" Sep 11 14:18:46.382222 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:46.382220 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="15fb504e-9451-46cf-a591-4ec1c0378ecc" containerName="istio-proxy" Sep 11 14:18:46.382315 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:46.382230 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="15fb504e-9451-46cf-a591-4ec1c0378ecc" containerName="istio-proxy" Sep 11 14:18:46.386714 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:46.386698 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k" Sep 11 14:18:46.390080 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:46.390058 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:18:46.396923 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:46.396893 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k"] Sep 11 14:18:46.462584 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:46.462543 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c579a3-e7f1-43e3-bff9-8a277467d859-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-20588104-h7c4k\" (UID: \"10c579a3-e7f1-43e3-bff9-8a277467d859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k" Sep 11 14:18:46.563886 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:46.563846 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c579a3-e7f1-43e3-bff9-8a277467d859-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-20588104-h7c4k\" (UID: \"10c579a3-e7f1-43e3-bff9-8a277467d859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k" Sep 11 14:18:46.564538 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:46.564519 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c579a3-e7f1-43e3-bff9-8a277467d859-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-20588104-h7c4k\" (UID: \"10c579a3-e7f1-43e3-bff9-8a277467d859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k" Sep 11 14:18:46.699401 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:46.699307 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k" Sep 11 14:18:46.995371 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:46.995342 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k"] Sep 11 14:18:46.996580 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:18:46.996534 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c579a3_e7f1_43e3_bff9_8a277467d859.slice/crio-4f53bff7c480a862fed1b44d1959ea56a0ca792dc70c3a44e345167188b57240 WatchSource:0}: Error finding container 4f53bff7c480a862fed1b44d1959ea56a0ca792dc70c3a44e345167188b57240: Status 404 returned error can't find the container with id 4f53bff7c480a862fed1b44d1959ea56a0ca792dc70c3a44e345167188b57240 Sep 11 14:18:47.283986 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:47.283895 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k" event={"ID":"10c579a3-e7f1-43e3-bff9-8a277467d859","Type":"ContainerStarted","Data":"994aa45202fd3011ba354947a0026fef5cd6457ea754442c81da82ffa292c0f1"} Sep 11 14:18:47.283986 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:47.283959 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k" event={"ID":"10c579a3-e7f1-43e3-bff9-8a277467d859","Type":"ContainerStarted","Data":"4f53bff7c480a862fed1b44d1959ea56a0ca792dc70c3a44e345167188b57240"} Sep 11 14:18:47.310810 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:47.310754 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k" podStartSLOduration=1.310737387 podStartE2EDuration="1.310737387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:47.308957954 +0000 UTC m=+1267.713939080" watchObservedRunningTime="2026-09-11 14:18:47.310737387 +0000 UTC m=+1267.715718525" Sep 11 14:18:48.291643 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:48.291610 2566 generic.go:356] "Generic (PLEG): container finished" podID="10c579a3-e7f1-43e3-bff9-8a277467d859" containerID="994aa45202fd3011ba354947a0026fef5cd6457ea754442c81da82ffa292c0f1" exitCode=0 Sep 11 14:18:48.292050 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:48.291698 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k" event={"ID":"10c579a3-e7f1-43e3-bff9-8a277467d859","Type":"ContainerDied","Data":"994aa45202fd3011ba354947a0026fef5cd6457ea754442c81da82ffa292c0f1"} Sep 11 14:18:49.439696 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:49.439670 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k" Sep 11 14:18:49.595474 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:49.595387 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c579a3-e7f1-43e3-bff9-8a277467d859-maas-api-service-ca\") pod \"10c579a3-e7f1-43e3-bff9-8a277467d859\" (UID: \"10c579a3-e7f1-43e3-bff9-8a277467d859\") " Sep 11 14:18:49.595840 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:49.595819 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c579a3-e7f1-43e3-bff9-8a277467d859-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10c579a3-e7f1-43e3-bff9-8a277467d859" (UID: "10c579a3-e7f1-43e3-bff9-8a277467d859"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:49.696658 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:49.696617 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c579a3-e7f1-43e3-bff9-8a277467d859-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:18:50.306618 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:50.306582 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k" Sep 11 14:18:50.306831 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:50.306622 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k" event={"ID":"10c579a3-e7f1-43e3-bff9-8a277467d859","Type":"ContainerDied","Data":"4f53bff7c480a862fed1b44d1959ea56a0ca792dc70c3a44e345167188b57240"} Sep 11 14:18:50.306831 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:50.306669 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f53bff7c480a862fed1b44d1959ea56a0ca792dc70c3a44e345167188b57240" Sep 11 14:18:57.652539 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:57.652501 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k"] Sep 11 14:18:57.663942 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:57.663901 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20588104-h7c4k"] Sep 11 14:18:58.098901 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:18:58.098858 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c579a3-e7f1-43e3-bff9-8a277467d859" path="/var/lib/kubelet/pods/10c579a3-e7f1-43e3-bff9-8a277467d859/volumes" Sep 11 14:19:05.251980 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:05.251949 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k"] Sep 11 14:19:05.252452 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:05.252220 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" podUID="c622c3b1-cddb-47e4-97e2-d5d67f858a2f" containerName="istio-proxy" containerID="cri-o://6bcf987564484e3212e35b988f1554b1701008cb2cc9316fca86ea7e140f742f" gracePeriod=30 Sep 11 14:19:07.797649 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:07.797611 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj"] Sep 11 14:19:07.798152 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:07.798050 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c579a3-e7f1-43e3-bff9-8a277467d859" containerName="revoke-keys" Sep 11 14:19:07.798152 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:07.798062 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c579a3-e7f1-43e3-bff9-8a277467d859" containerName="revoke-keys" Sep 11 14:19:07.798292 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:07.798183 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c579a3-e7f1-43e3-bff9-8a277467d859" containerName="revoke-keys" Sep 11 14:19:07.801488 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:07.801468 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj" Sep 11 14:19:07.806177 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:07.806149 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:19:07.814754 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:07.814733 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj"] Sep 11 14:19:07.874502 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:07.874473 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957ae00e-4e4b-42b5-bd1d-e976e68da812-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj\" (UID: \"957ae00e-4e4b-42b5-bd1d-e976e68da812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj" Sep 11 14:19:07.975620 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:07.975585 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957ae00e-4e4b-42b5-bd1d-e976e68da812-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj\" (UID: \"957ae00e-4e4b-42b5-bd1d-e976e68da812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj" Sep 11 14:19:07.976224 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:07.976201 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957ae00e-4e4b-42b5-bd1d-e976e68da812-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj\" (UID: \"957ae00e-4e4b-42b5-bd1d-e976e68da812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj" Sep 11 14:19:08.112836 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:08.112750 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj" Sep 11 14:19:08.393403 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:08.393360 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj"] Sep 11 14:19:08.398606 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:19:08.398409 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957ae00e_4e4b_42b5_bd1d_e976e68da812.slice/crio-2ee596992faf7cfb70f1f91ed369034606c7afd437f0134102ca180f502aae42 WatchSource:0}: Error finding container 2ee596992faf7cfb70f1f91ed369034606c7afd437f0134102ca180f502aae42: Status 404 returned error can't find the container with id 2ee596992faf7cfb70f1f91ed369034606c7afd437f0134102ca180f502aae42 Sep 11 14:19:08.429003 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:08.428970 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj" event={"ID":"957ae00e-4e4b-42b5-bd1d-e976e68da812","Type":"ContainerStarted","Data":"2ee596992faf7cfb70f1f91ed369034606c7afd437f0134102ca180f502aae42"} Sep 11 14:19:09.223498 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:09.223464 2566 patch_prober.go:28] interesting pod/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:19:09.223868 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:09.223526 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" podUID="c622c3b1-cddb-47e4-97e2-d5d67f858a2f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:19:09.436622 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:09.436536 2566 generic.go:356] "Generic (PLEG): container finished" podID="957ae00e-4e4b-42b5-bd1d-e976e68da812" containerID="bf4683252b052fc09e430fc7920a871d015e0477f287440a13740432cb84d1d1" exitCode=0 Sep 11 14:19:09.436766 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:09.436625 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj" event={"ID":"957ae00e-4e4b-42b5-bd1d-e976e68da812","Type":"ContainerDied","Data":"bf4683252b052fc09e430fc7920a871d015e0477f287440a13740432cb84d1d1"} Sep 11 14:19:10.446233 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.446201 2566 generic.go:356] "Generic (PLEG): container finished" podID="c622c3b1-cddb-47e4-97e2-d5d67f858a2f" containerID="6bcf987564484e3212e35b988f1554b1701008cb2cc9316fca86ea7e140f742f" exitCode=0 Sep 11 14:19:10.446683 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.446274 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" event={"ID":"c622c3b1-cddb-47e4-97e2-d5d67f858a2f","Type":"ContainerDied","Data":"6bcf987564484e3212e35b988f1554b1701008cb2cc9316fca86ea7e140f742f"} Sep 11 14:19:10.523592 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.523537 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:19:10.600702 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.600677 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-credential-socket\") pod \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " Sep 11 14:19:10.600846 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.600714 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-envoy\") pod \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " Sep 11 14:19:10.600846 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.600762 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-workload-socket\") pod \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " Sep 11 14:19:10.600846 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.600800 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-data\") pod \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " Sep 11 14:19:10.601032 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.600859 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-workload-certs\") pod \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " Sep 11 14:19:10.601032 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.600899 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7s9r\" (UniqueName: \"kubernetes.io/projected/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-kube-api-access-k7s9r\") pod \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " Sep 11 14:19:10.601032 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.600949 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istiod-ca-cert\") pod \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " Sep 11 14:19:10.601032 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.600985 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-podinfo\") pod \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " Sep 11 14:19:10.601032 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.600983 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c622c3b1-cddb-47e4-97e2-d5d67f858a2f" (UID: "c622c3b1-cddb-47e4-97e2-d5d67f858a2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:10.601032 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.601017 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-token\") pod \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\" (UID: \"c622c3b1-cddb-47e4-97e2-d5d67f858a2f\") " Sep 11 14:19:10.601377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.601350 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c622c3b1-cddb-47e4-97e2-d5d67f858a2f" (UID: "c622c3b1-cddb-47e4-97e2-d5d67f858a2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:10.601509 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.601454 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:10.601509 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.601482 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:10.601810 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.601701 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c622c3b1-cddb-47e4-97e2-d5d67f858a2f" (UID: "c622c3b1-cddb-47e4-97e2-d5d67f858a2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:10.601872 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.601848 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c622c3b1-cddb-47e4-97e2-d5d67f858a2f" (UID: "c622c3b1-cddb-47e4-97e2-d5d67f858a2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:19:10.602222 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.602196 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-data" (OuterVolumeSpecName: "istio-data") pod "c622c3b1-cddb-47e4-97e2-d5d67f858a2f" (UID: "c622c3b1-cddb-47e4-97e2-d5d67f858a2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:10.603467 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.603203 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj" Sep 11 14:19:10.603467 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.603344 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c622c3b1-cddb-47e4-97e2-d5d67f858a2f" (UID: "c622c3b1-cddb-47e4-97e2-d5d67f858a2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:10.603799 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.603776 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "c622c3b1-cddb-47e4-97e2-d5d67f858a2f" (UID: "c622c3b1-cddb-47e4-97e2-d5d67f858a2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:10.604294 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.604274 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c622c3b1-cddb-47e4-97e2-d5d67f858a2f" (UID: "c622c3b1-cddb-47e4-97e2-d5d67f858a2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:19:10.604449 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.604426 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-kube-api-access-k7s9r" (OuterVolumeSpecName: "kube-api-access-k7s9r") pod "c622c3b1-cddb-47e4-97e2-d5d67f858a2f" (UID: "c622c3b1-cddb-47e4-97e2-d5d67f858a2f"). InnerVolumeSpecName "kube-api-access-k7s9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:10.702581 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.702546 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957ae00e-4e4b-42b5-bd1d-e976e68da812-maas-api-service-ca\") pod \"957ae00e-4e4b-42b5-bd1d-e976e68da812\" (UID: \"957ae00e-4e4b-42b5-bd1d-e976e68da812\") " Sep 11 14:19:10.702878 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.702847 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957ae00e-4e4b-42b5-bd1d-e976e68da812-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "957ae00e-4e4b-42b5-bd1d-e976e68da812" (UID: "957ae00e-4e4b-42b5-bd1d-e976e68da812"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:19:10.702956 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.702939 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:10.702995 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.702962 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7s9r\" (UniqueName: \"kubernetes.io/projected/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-kube-api-access-k7s9r\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:10.702995 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.702977 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:10.702995 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.702990 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:10.703107 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.703001 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:10.703107 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.703013 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:10.703107 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.703025 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/957ae00e-4e4b-42b5-bd1d-e976e68da812-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:10.703107 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:10.703037 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c622c3b1-cddb-47e4-97e2-d5d67f858a2f-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:11.454156 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:11.454116 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" event={"ID":"c622c3b1-cddb-47e4-97e2-d5d67f858a2f","Type":"ContainerDied","Data":"e54e2e83d3e460579c549f464f08ce362b0948c0666d7f5c2bbf0f051a8e841a"} Sep 11 14:19:11.454858 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:11.454160 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k" Sep 11 14:19:11.454858 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:11.454193 2566 scope.go:117] "RemoveContainer" containerID="6bcf987564484e3212e35b988f1554b1701008cb2cc9316fca86ea7e140f742f" Sep 11 14:19:11.455710 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:11.455677 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj" event={"ID":"957ae00e-4e4b-42b5-bd1d-e976e68da812","Type":"ContainerDied","Data":"2ee596992faf7cfb70f1f91ed369034606c7afd437f0134102ca180f502aae42"} Sep 11 14:19:11.455710 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:11.455698 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj" Sep 11 14:19:11.455864 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:11.455737 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ee596992faf7cfb70f1f91ed369034606c7afd437f0134102ca180f502aae42" Sep 11 14:19:11.486150 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:11.486071 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k"] Sep 11 14:19:11.497726 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:11.497695 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-ea85ff-openshift-default-7f5c9d6f8b-42s2k"] Sep 11 14:19:12.084986 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:12.084957 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277"] Sep 11 14:19:12.085193 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:12.085172 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" podUID="410f7071-8ff3-44d6-84a7-921ad9624ff9" containerName="istio-proxy" containerID="cri-o://e0426f1639d4f78ba9dd5d6fdebc2e2ae8ebce06a8634a707fe1693799a21212" gracePeriod=30 Sep 11 14:19:12.097841 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:12.097811 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c622c3b1-cddb-47e4-97e2-d5d67f858a2f" path="/var/lib/kubelet/pods/c622c3b1-cddb-47e4-97e2-d5d67f858a2f/volumes" Sep 11 14:19:17.348222 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.348159 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:19:17.472184 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.472144 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-token\") pod \"410f7071-8ff3-44d6-84a7-921ad9624ff9\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " Sep 11 14:19:17.472383 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.472227 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-podinfo\") pod \"410f7071-8ff3-44d6-84a7-921ad9624ff9\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " Sep 11 14:19:17.472383 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.472256 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-credential-socket\") pod \"410f7071-8ff3-44d6-84a7-921ad9624ff9\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " Sep 11 14:19:17.472383 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.472289 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-workload-certs\") pod \"410f7071-8ff3-44d6-84a7-921ad9624ff9\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " Sep 11 14:19:17.472383 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.472313 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-data\") pod \"410f7071-8ff3-44d6-84a7-921ad9624ff9\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " Sep 11 14:19:17.472595 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.472392 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-workload-socket\") pod \"410f7071-8ff3-44d6-84a7-921ad9624ff9\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " Sep 11 14:19:17.472595 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.472439 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-envoy\") pod \"410f7071-8ff3-44d6-84a7-921ad9624ff9\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " Sep 11 14:19:17.472595 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.472473 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px9g4\" (UniqueName: \"kubernetes.io/projected/410f7071-8ff3-44d6-84a7-921ad9624ff9-kube-api-access-px9g4\") pod \"410f7071-8ff3-44d6-84a7-921ad9624ff9\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " Sep 11 14:19:17.472595 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.472567 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/410f7071-8ff3-44d6-84a7-921ad9624ff9-istiod-ca-cert\") pod \"410f7071-8ff3-44d6-84a7-921ad9624ff9\" (UID: \"410f7071-8ff3-44d6-84a7-921ad9624ff9\") " Sep 11 14:19:17.472790 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.472635 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "410f7071-8ff3-44d6-84a7-921ad9624ff9" (UID: "410f7071-8ff3-44d6-84a7-921ad9624ff9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:17.472790 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.472645 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "410f7071-8ff3-44d6-84a7-921ad9624ff9" (UID: "410f7071-8ff3-44d6-84a7-921ad9624ff9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:17.472790 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.472660 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-data" (OuterVolumeSpecName: "istio-data") pod "410f7071-8ff3-44d6-84a7-921ad9624ff9" (UID: "410f7071-8ff3-44d6-84a7-921ad9624ff9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:17.473110 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.472956 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "410f7071-8ff3-44d6-84a7-921ad9624ff9" (UID: "410f7071-8ff3-44d6-84a7-921ad9624ff9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:17.473209 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.473141 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:17.473209 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.473161 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:17.473209 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.473032 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/410f7071-8ff3-44d6-84a7-921ad9624ff9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "410f7071-8ff3-44d6-84a7-921ad9624ff9" (UID: "410f7071-8ff3-44d6-84a7-921ad9624ff9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:19:17.473209 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.473176 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:17.474582 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.474559 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-token" (OuterVolumeSpecName: "istio-token") pod "410f7071-8ff3-44d6-84a7-921ad9624ff9" (UID: "410f7071-8ff3-44d6-84a7-921ad9624ff9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:17.474676 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.474637 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "410f7071-8ff3-44d6-84a7-921ad9624ff9" (UID: "410f7071-8ff3-44d6-84a7-921ad9624ff9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:19:17.474860 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.474844 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/410f7071-8ff3-44d6-84a7-921ad9624ff9-kube-api-access-px9g4" (OuterVolumeSpecName: "kube-api-access-px9g4") pod "410f7071-8ff3-44d6-84a7-921ad9624ff9" (UID: "410f7071-8ff3-44d6-84a7-921ad9624ff9"). InnerVolumeSpecName "kube-api-access-px9g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:17.474924 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.474911 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "410f7071-8ff3-44d6-84a7-921ad9624ff9" (UID: "410f7071-8ff3-44d6-84a7-921ad9624ff9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:17.506474 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.506439 2566 generic.go:356] "Generic (PLEG): container finished" podID="410f7071-8ff3-44d6-84a7-921ad9624ff9" containerID="e0426f1639d4f78ba9dd5d6fdebc2e2ae8ebce06a8634a707fe1693799a21212" exitCode=0 Sep 11 14:19:17.506639 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.506486 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" Sep 11 14:19:17.506639 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.506523 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" event={"ID":"410f7071-8ff3-44d6-84a7-921ad9624ff9","Type":"ContainerDied","Data":"e0426f1639d4f78ba9dd5d6fdebc2e2ae8ebce06a8634a707fe1693799a21212"} Sep 11 14:19:17.506639 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.506583 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277" event={"ID":"410f7071-8ff3-44d6-84a7-921ad9624ff9","Type":"ContainerDied","Data":"793a6048a1e34a49f3f71451ae42df92b8573b4a4082282d11889b27398d8d43"} Sep 11 14:19:17.506639 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.506612 2566 scope.go:117] "RemoveContainer" containerID="e0426f1639d4f78ba9dd5d6fdebc2e2ae8ebce06a8634a707fe1693799a21212" Sep 11 14:19:17.523755 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.523733 2566 scope.go:117] "RemoveContainer" containerID="e7c8a547b649cab07930674c89047387ce7b0420be9f0090a9bf658c13a9e3e0" Sep 11 14:19:17.532952 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.532932 2566 scope.go:117] "RemoveContainer" containerID="e0426f1639d4f78ba9dd5d6fdebc2e2ae8ebce06a8634a707fe1693799a21212" Sep 11 14:19:17.533261 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:19:17.533242 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0426f1639d4f78ba9dd5d6fdebc2e2ae8ebce06a8634a707fe1693799a21212\": container with ID starting with e0426f1639d4f78ba9dd5d6fdebc2e2ae8ebce06a8634a707fe1693799a21212 not found: ID does not exist" containerID="e0426f1639d4f78ba9dd5d6fdebc2e2ae8ebce06a8634a707fe1693799a21212" Sep 11 14:19:17.533306 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.533273 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0426f1639d4f78ba9dd5d6fdebc2e2ae8ebce06a8634a707fe1693799a21212"} err="failed to get container status \"e0426f1639d4f78ba9dd5d6fdebc2e2ae8ebce06a8634a707fe1693799a21212\": rpc error: code = NotFound desc = could not find container \"e0426f1639d4f78ba9dd5d6fdebc2e2ae8ebce06a8634a707fe1693799a21212\": container with ID starting with e0426f1639d4f78ba9dd5d6fdebc2e2ae8ebce06a8634a707fe1693799a21212 not found: ID does not exist" Sep 11 14:19:17.533306 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.533290 2566 scope.go:117] "RemoveContainer" containerID="e7c8a547b649cab07930674c89047387ce7b0420be9f0090a9bf658c13a9e3e0" Sep 11 14:19:17.533547 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:19:17.533527 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c8a547b649cab07930674c89047387ce7b0420be9f0090a9bf658c13a9e3e0\": container with ID starting with e7c8a547b649cab07930674c89047387ce7b0420be9f0090a9bf658c13a9e3e0 not found: ID does not exist" containerID="e7c8a547b649cab07930674c89047387ce7b0420be9f0090a9bf658c13a9e3e0" Sep 11 14:19:17.533615 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.533556 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c8a547b649cab07930674c89047387ce7b0420be9f0090a9bf658c13a9e3e0"} err="failed to get container status \"e7c8a547b649cab07930674c89047387ce7b0420be9f0090a9bf658c13a9e3e0\": rpc error: code = NotFound desc = could not find container \"e7c8a547b649cab07930674c89047387ce7b0420be9f0090a9bf658c13a9e3e0\": container with ID starting with e7c8a547b649cab07930674c89047387ce7b0420be9f0090a9bf658c13a9e3e0 not found: ID does not exist" Sep 11 14:19:17.538221 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.538200 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277"] Sep 11 14:19:17.552450 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.552430 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-20588104-openshift-default-7fd8d85bcb-wz277"] Sep 11 14:19:17.574372 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.574350 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:17.574372 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.574372 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:17.574510 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.574381 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:17.574510 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.574389 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/410f7071-8ff3-44d6-84a7-921ad9624ff9-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:17.574510 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.574398 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px9g4\" (UniqueName: \"kubernetes.io/projected/410f7071-8ff3-44d6-84a7-921ad9624ff9-kube-api-access-px9g4\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:17.574510 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:17.574406 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/410f7071-8ff3-44d6-84a7-921ad9624ff9-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:18.098334 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:18.098300 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="410f7071-8ff3-44d6-84a7-921ad9624ff9" path="/var/lib/kubelet/pods/410f7071-8ff3-44d6-84a7-921ad9624ff9/volumes" Sep 11 14:19:18.681507 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:18.681469 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be0c80-29818935-nmnb4"] Sep 11 14:19:18.688256 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:18.688228 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-be0c80-29818935-nmnb4"] Sep 11 14:19:19.027404 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:19.027375 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj"] Sep 11 14:19:19.029770 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:19.029752 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-be0c80-dlvcj"] Sep 11 14:19:20.098332 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:20.098295 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="957ae00e-4e4b-42b5-bd1d-e976e68da812" path="/var/lib/kubelet/pods/957ae00e-4e4b-42b5-bd1d-e976e68da812/volumes" Sep 11 14:19:20.098785 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:20.098557 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfafad60-2828-4eed-a741-49f914583bef" path="/var/lib/kubelet/pods/bfafad60-2828-4eed-a741-49f914583bef/volumes" Sep 11 14:19:26.145065 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.145034 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8"] Sep 11 14:19:26.145497 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.145280 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" podUID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" containerName="istio-proxy" containerID="cri-o://17cda0f3900842eef6ccf895cb567409d84f563c1f925bc51c43c2fb0f4edccc" gracePeriod=30 Sep 11 14:19:26.532649 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.532614 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt"] Sep 11 14:19:26.533034 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.533021 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c622c3b1-cddb-47e4-97e2-d5d67f858a2f" containerName="istio-proxy" Sep 11 14:19:26.533098 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.533036 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c622c3b1-cddb-47e4-97e2-d5d67f858a2f" containerName="istio-proxy" Sep 11 14:19:26.533098 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.533055 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="410f7071-8ff3-44d6-84a7-921ad9624ff9" containerName="istio-proxy" Sep 11 14:19:26.533098 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.533061 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="410f7071-8ff3-44d6-84a7-921ad9624ff9" containerName="istio-proxy" Sep 11 14:19:26.533098 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.533066 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="410f7071-8ff3-44d6-84a7-921ad9624ff9" containerName="istio-proxy" Sep 11 14:19:26.533098 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.533072 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="410f7071-8ff3-44d6-84a7-921ad9624ff9" containerName="istio-proxy" Sep 11 14:19:26.533098 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.533080 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957ae00e-4e4b-42b5-bd1d-e976e68da812" containerName="revoke-keys" Sep 11 14:19:26.533098 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.533098 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="957ae00e-4e4b-42b5-bd1d-e976e68da812" containerName="revoke-keys" Sep 11 14:19:26.533319 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.533155 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="957ae00e-4e4b-42b5-bd1d-e976e68da812" containerName="revoke-keys" Sep 11 14:19:26.533319 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.533163 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="410f7071-8ff3-44d6-84a7-921ad9624ff9" containerName="istio-proxy" Sep 11 14:19:26.533319 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.533172 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="410f7071-8ff3-44d6-84a7-921ad9624ff9" containerName="istio-proxy" Sep 11 14:19:26.533319 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.533178 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c622c3b1-cddb-47e4-97e2-d5d67f858a2f" containerName="istio-proxy" Sep 11 14:19:26.537868 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.537846 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt" Sep 11 14:19:26.540984 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.540964 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:19:26.554067 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.554043 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt"] Sep 11 14:19:26.663301 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.663266 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0feb9a9b-1bd1-405a-a65f-e7b15b68de36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt\" (UID: \"0feb9a9b-1bd1-405a-a65f-e7b15b68de36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt" Sep 11 14:19:26.764154 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.764078 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0feb9a9b-1bd1-405a-a65f-e7b15b68de36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt\" (UID: \"0feb9a9b-1bd1-405a-a65f-e7b15b68de36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt" Sep 11 14:19:26.764697 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.764676 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0feb9a9b-1bd1-405a-a65f-e7b15b68de36-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt\" (UID: \"0feb9a9b-1bd1-405a-a65f-e7b15b68de36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt" Sep 11 14:19:26.848310 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:26.848222 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt" Sep 11 14:19:27.127788 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:27.127756 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt"] Sep 11 14:19:27.128494 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:19:27.128462 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0feb9a9b_1bd1_405a_a65f_e7b15b68de36.slice/crio-90a6f61d53c5dacdcd71d338c234f254ffe17f089a2f6bd52349689ecc8c7ff0 WatchSource:0}: Error finding container 90a6f61d53c5dacdcd71d338c234f254ffe17f089a2f6bd52349689ecc8c7ff0: Status 404 returned error can't find the container with id 90a6f61d53c5dacdcd71d338c234f254ffe17f089a2f6bd52349689ecc8c7ff0 Sep 11 14:19:27.573102 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:27.573052 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt" event={"ID":"0feb9a9b-1bd1-405a-a65f-e7b15b68de36","Type":"ContainerStarted","Data":"acde355de24d22baa142abbf7caffc58a19ebfdd466f36c5f19fb5b94d3355c2"} Sep 11 14:19:27.573536 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:27.573118 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt" event={"ID":"0feb9a9b-1bd1-405a-a65f-e7b15b68de36","Type":"ContainerStarted","Data":"90a6f61d53c5dacdcd71d338c234f254ffe17f089a2f6bd52349689ecc8c7ff0"} Sep 11 14:19:27.593773 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:27.593724 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt" podStartSLOduration=1.593708363 podStartE2EDuration="1.593708363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:27.590431524 +0000 UTC m=+1307.995412663" watchObservedRunningTime="2026-09-11 14:19:27.593708363 +0000 UTC m=+1307.998689491" Sep 11 14:19:28.581283 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:28.581245 2566 generic.go:356] "Generic (PLEG): container finished" podID="0feb9a9b-1bd1-405a-a65f-e7b15b68de36" containerID="acde355de24d22baa142abbf7caffc58a19ebfdd466f36c5f19fb5b94d3355c2" exitCode=0 Sep 11 14:19:28.581680 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:28.581334 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt" event={"ID":"0feb9a9b-1bd1-405a-a65f-e7b15b68de36","Type":"ContainerDied","Data":"acde355de24d22baa142abbf7caffc58a19ebfdd466f36c5f19fb5b94d3355c2"} Sep 11 14:19:29.749106 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:29.749055 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt" Sep 11 14:19:29.893520 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:29.893437 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0feb9a9b-1bd1-405a-a65f-e7b15b68de36-maas-api-service-ca\") pod \"0feb9a9b-1bd1-405a-a65f-e7b15b68de36\" (UID: \"0feb9a9b-1bd1-405a-a65f-e7b15b68de36\") " Sep 11 14:19:29.893769 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:29.893745 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0feb9a9b-1bd1-405a-a65f-e7b15b68de36-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0feb9a9b-1bd1-405a-a65f-e7b15b68de36" (UID: "0feb9a9b-1bd1-405a-a65f-e7b15b68de36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:19:29.994228 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:29.994187 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0feb9a9b-1bd1-405a-a65f-e7b15b68de36-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:30.597192 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.597165 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt" Sep 11 14:19:30.597911 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.597188 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt" event={"ID":"0feb9a9b-1bd1-405a-a65f-e7b15b68de36","Type":"ContainerDied","Data":"90a6f61d53c5dacdcd71d338c234f254ffe17f089a2f6bd52349689ecc8c7ff0"} Sep 11 14:19:30.598118 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.598070 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a6f61d53c5dacdcd71d338c234f254ffe17f089a2f6bd52349689ecc8c7ff0" Sep 11 14:19:30.639778 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.639748 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb"] Sep 11 14:19:30.640173 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.640159 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0feb9a9b-1bd1-405a-a65f-e7b15b68de36" containerName="revoke-keys" Sep 11 14:19:30.640225 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.640174 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0feb9a9b-1bd1-405a-a65f-e7b15b68de36" containerName="revoke-keys" Sep 11 14:19:30.640262 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.640250 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0feb9a9b-1bd1-405a-a65f-e7b15b68de36" containerName="revoke-keys" Sep 11 14:19:30.643310 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.643293 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.646800 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.646781 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-fb993c-openshift-default-dockercfg-2n2zg\"" Sep 11 14:19:30.668703 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.668679 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb"] Sep 11 14:19:30.802518 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.802483 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8smg8\" (UniqueName: \"kubernetes.io/projected/a5f713d3-d284-4778-b474-1df1a1d06d17-kube-api-access-8smg8\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.802904 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.802532 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-credential-socket\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.802904 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.802609 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-data\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.802904 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.802633 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-workload-socket\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.802904 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.802751 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f713d3-d284-4778-b474-1df1a1d06d17-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.802904 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.802796 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-workload-certs\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.802904 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.802821 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-envoy\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.802904 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.802844 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-podinfo\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.802904 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.802872 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-token\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.903482 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.903396 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f713d3-d284-4778-b474-1df1a1d06d17-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.903482 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.903432 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-workload-certs\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.903482 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.903451 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-envoy\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.903482 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.903477 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-podinfo\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.903817 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.903666 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-token\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.903817 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.903749 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8smg8\" (UniqueName: \"kubernetes.io/projected/a5f713d3-d284-4778-b474-1df1a1d06d17-kube-api-access-8smg8\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.903817 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.903804 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-credential-socket\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.903969 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.903895 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-data\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.903969 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.903922 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-workload-socket\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.903969 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.903934 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-workload-certs\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.904293 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.904272 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-workload-socket\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.904391 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.904312 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-credential-socket\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.904391 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.904365 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-data\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.905010 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.904986 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f713d3-d284-4778-b474-1df1a1d06d17-istiod-ca-cert\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.906398 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.906380 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-podinfo\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.906576 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.906556 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-envoy\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.912218 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.912194 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-token\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.912663 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.912643 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8smg8\" (UniqueName: \"kubernetes.io/projected/a5f713d3-d284-4778-b474-1df1a1d06d17-kube-api-access-8smg8\") pod \"e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:30.953679 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:30.953641 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:31.244224 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.244199 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb"] Sep 11 14:19:31.246009 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:19:31.245981 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f713d3_d284_4778_b474_1df1a1d06d17.slice/crio-260f90a63ea20ace76ce028c1868bbc251fd9b4f321a6b30c3cc2346d665c6ed WatchSource:0}: Error finding container 260f90a63ea20ace76ce028c1868bbc251fd9b4f321a6b30c3cc2346d665c6ed: Status 404 returned error can't find the container with id 260f90a63ea20ace76ce028c1868bbc251fd9b4f321a6b30c3cc2346d665c6ed Sep 11 14:19:31.249949 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.249920 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:19:31.250048 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.249978 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:19:31.250048 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.250008 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:19:31.402698 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.402672 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:19:31.510835 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.510801 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-data\") pod \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " Sep 11 14:19:31.511041 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.510883 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-workload-certs\") pod \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " Sep 11 14:19:31.511041 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.510922 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-credential-socket\") pod \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " Sep 11 14:19:31.511041 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.510957 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xwjg\" (UniqueName: \"kubernetes.io/projected/fffb9f82-116a-4c91-bb26-3fd11261e4b8-kube-api-access-5xwjg\") pod \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " Sep 11 14:19:31.511041 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.510977 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-podinfo\") pod \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " Sep 11 14:19:31.511041 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.511009 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istiod-ca-cert\") pod \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " Sep 11 14:19:31.511041 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.511026 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-token\") pod \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " Sep 11 14:19:31.511041 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.511042 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-workload-socket\") pod \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " Sep 11 14:19:31.511444 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.511045 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "fffb9f82-116a-4c91-bb26-3fd11261e4b8" (UID: "fffb9f82-116a-4c91-bb26-3fd11261e4b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:31.511444 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.511099 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-envoy\") pod \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\" (UID: \"fffb9f82-116a-4c91-bb26-3fd11261e4b8\") " Sep 11 14:19:31.511444 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.511113 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fffb9f82-116a-4c91-bb26-3fd11261e4b8" (UID: "fffb9f82-116a-4c91-bb26-3fd11261e4b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:31.511444 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.511217 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fffb9f82-116a-4c91-bb26-3fd11261e4b8" (UID: "fffb9f82-116a-4c91-bb26-3fd11261e4b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:31.511444 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.511288 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fffb9f82-116a-4c91-bb26-3fd11261e4b8" (UID: "fffb9f82-116a-4c91-bb26-3fd11261e4b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:31.511444 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.511385 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fffb9f82-116a-4c91-bb26-3fd11261e4b8" (UID: "fffb9f82-116a-4c91-bb26-3fd11261e4b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:19:31.511755 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.511674 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:31.511755 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.511693 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:31.511755 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.511706 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:31.511755 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.511719 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:31.511755 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.511731 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:31.513558 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.513524 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fffb9f82-116a-4c91-bb26-3fd11261e4b8" (UID: "fffb9f82-116a-4c91-bb26-3fd11261e4b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:31.513688 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.513573 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffb9f82-116a-4c91-bb26-3fd11261e4b8-kube-api-access-5xwjg" (OuterVolumeSpecName: "kube-api-access-5xwjg") pod "fffb9f82-116a-4c91-bb26-3fd11261e4b8" (UID: "fffb9f82-116a-4c91-bb26-3fd11261e4b8"). InnerVolumeSpecName "kube-api-access-5xwjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:31.513927 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.513905 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "fffb9f82-116a-4c91-bb26-3fd11261e4b8" (UID: "fffb9f82-116a-4c91-bb26-3fd11261e4b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:31.513927 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.513911 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fffb9f82-116a-4c91-bb26-3fd11261e4b8" (UID: "fffb9f82-116a-4c91-bb26-3fd11261e4b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:19:31.607824 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.607782 2566 generic.go:356] "Generic (PLEG): container finished" podID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" containerID="17cda0f3900842eef6ccf895cb567409d84f563c1f925bc51c43c2fb0f4edccc" exitCode=0 Sep 11 14:19:31.607996 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.607835 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" Sep 11 14:19:31.607996 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.607870 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" event={"ID":"fffb9f82-116a-4c91-bb26-3fd11261e4b8","Type":"ContainerDied","Data":"17cda0f3900842eef6ccf895cb567409d84f563c1f925bc51c43c2fb0f4edccc"} Sep 11 14:19:31.607996 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.607940 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8" event={"ID":"fffb9f82-116a-4c91-bb26-3fd11261e4b8","Type":"ContainerDied","Data":"2ca6445a0db2a3bfcb9a3218b48ab6765870ade3911394f274a66ef5240ae79e"} Sep 11 14:19:31.607996 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.607974 2566 scope.go:117] "RemoveContainer" containerID="17cda0f3900842eef6ccf895cb567409d84f563c1f925bc51c43c2fb0f4edccc" Sep 11 14:19:31.609824 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.609799 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" event={"ID":"a5f713d3-d284-4778-b474-1df1a1d06d17","Type":"ContainerStarted","Data":"a26be4307612c0e28d6434346cae6b5f33ecb22a415a72d3fb134c8a0588c1fe"} Sep 11 14:19:31.609946 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.609848 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" event={"ID":"a5f713d3-d284-4778-b474-1df1a1d06d17","Type":"ContainerStarted","Data":"260f90a63ea20ace76ce028c1868bbc251fd9b4f321a6b30c3cc2346d665c6ed"} Sep 11 14:19:31.612254 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.612226 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xwjg\" (UniqueName: \"kubernetes.io/projected/fffb9f82-116a-4c91-bb26-3fd11261e4b8-kube-api-access-5xwjg\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:31.612254 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.612252 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:31.612410 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.612263 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:31.612410 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.612274 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fffb9f82-116a-4c91-bb26-3fd11261e4b8-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:31.618888 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.618866 2566 scope.go:117] "RemoveContainer" containerID="84b1bfd6177963b651d10d9b71b716056acad4d6eaa8739e8cfcc5fbb89cda23" Sep 11 14:19:31.627663 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.627634 2566 scope.go:117] "RemoveContainer" containerID="17cda0f3900842eef6ccf895cb567409d84f563c1f925bc51c43c2fb0f4edccc" Sep 11 14:19:31.627921 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:19:31.627902 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17cda0f3900842eef6ccf895cb567409d84f563c1f925bc51c43c2fb0f4edccc\": container with ID starting with 17cda0f3900842eef6ccf895cb567409d84f563c1f925bc51c43c2fb0f4edccc not found: ID does not exist" containerID="17cda0f3900842eef6ccf895cb567409d84f563c1f925bc51c43c2fb0f4edccc" Sep 11 14:19:31.627967 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.627935 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17cda0f3900842eef6ccf895cb567409d84f563c1f925bc51c43c2fb0f4edccc"} err="failed to get container status \"17cda0f3900842eef6ccf895cb567409d84f563c1f925bc51c43c2fb0f4edccc\": rpc error: code = NotFound desc = could not find container \"17cda0f3900842eef6ccf895cb567409d84f563c1f925bc51c43c2fb0f4edccc\": container with ID starting with 17cda0f3900842eef6ccf895cb567409d84f563c1f925bc51c43c2fb0f4edccc not found: ID does not exist" Sep 11 14:19:31.627967 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.627954 2566 scope.go:117] "RemoveContainer" containerID="84b1bfd6177963b651d10d9b71b716056acad4d6eaa8739e8cfcc5fbb89cda23" Sep 11 14:19:31.628235 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:19:31.628217 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b1bfd6177963b651d10d9b71b716056acad4d6eaa8739e8cfcc5fbb89cda23\": container with ID starting with 84b1bfd6177963b651d10d9b71b716056acad4d6eaa8739e8cfcc5fbb89cda23 not found: ID does not exist" containerID="84b1bfd6177963b651d10d9b71b716056acad4d6eaa8739e8cfcc5fbb89cda23" Sep 11 14:19:31.628279 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.628245 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b1bfd6177963b651d10d9b71b716056acad4d6eaa8739e8cfcc5fbb89cda23"} err="failed to get container status \"84b1bfd6177963b651d10d9b71b716056acad4d6eaa8739e8cfcc5fbb89cda23\": rpc error: code = NotFound desc = could not find container \"84b1bfd6177963b651d10d9b71b716056acad4d6eaa8739e8cfcc5fbb89cda23\": container with ID starting with 84b1bfd6177963b651d10d9b71b716056acad4d6eaa8739e8cfcc5fbb89cda23 not found: ID does not exist" Sep 11 14:19:31.639206 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.639159 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" podStartSLOduration=1.639138757 podStartE2EDuration="1.639138757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:31.635889991 +0000 UTC m=+1312.040871120" watchObservedRunningTime="2026-09-11 14:19:31.639138757 +0000 UTC m=+1312.044119886" Sep 11 14:19:31.656304 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.656277 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8"] Sep 11 14:19:31.674249 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.674223 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-be0c80-openshift-default-69bb677578-klxb8"] Sep 11 14:19:31.954700 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:31.954668 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:32.099285 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:32.099250 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" path="/var/lib/kubelet/pods/fffb9f82-116a-4c91-bb26-3fd11261e4b8/volumes" Sep 11 14:19:32.959908 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:32.959875 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:33.624517 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:33.624478 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:33.625674 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:33.625653 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:37.940321 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:37.940289 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt"] Sep 11 14:19:37.944782 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:37.944757 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c6156e-dmlmt"] Sep 11 14:19:38.098800 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:38.098756 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0feb9a9b-1bd1-405a-a65f-e7b15b68de36" path="/var/lib/kubelet/pods/0feb9a9b-1bd1-405a-a65f-e7b15b68de36/volumes" Sep 11 14:19:46.044068 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:46.044028 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55"] Sep 11 14:19:46.044563 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:46.044265 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" podUID="5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" containerName="istio-proxy" containerID="cri-o://f8373d228c54125a4ed5d1dfa09f501dbeece8f3f5e3840b0bc487bba2f3aebf" gracePeriod=30 Sep 11 14:19:47.070207 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.070170 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw"] Sep 11 14:19:47.070673 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.070657 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" containerName="istio-proxy" Sep 11 14:19:47.070720 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.070676 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" containerName="istio-proxy" Sep 11 14:19:47.070762 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.070753 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" containerName="istio-proxy" Sep 11 14:19:47.070801 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.070764 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" containerName="istio-proxy" Sep 11 14:19:47.075589 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.075572 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.087335 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.087317 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-1e2e7d-openshift-default-dockercfg-b6mxt\"" Sep 11 14:19:47.161219 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.161185 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw"] Sep 11 14:19:47.264944 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.264892 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qk4s\" (UniqueName: \"kubernetes.io/projected/8849f8ab-2b78-439c-911c-323c8c21c8b3-kube-api-access-8qk4s\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.265164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.264979 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-podinfo\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.265164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.265045 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-workload-certs\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.265164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.265077 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8849f8ab-2b78-439c-911c-323c8c21c8b3-istiod-ca-cert\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.265164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.265137 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-data\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.265352 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.265174 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-token\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.265352 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.265194 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-workload-socket\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.265352 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.265224 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-credential-socket\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.265352 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.265239 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-envoy\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.366513 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.366412 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-credential-socket\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.366513 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.366449 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-envoy\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.366513 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.366475 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qk4s\" (UniqueName: \"kubernetes.io/projected/8849f8ab-2b78-439c-911c-323c8c21c8b3-kube-api-access-8qk4s\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.366780 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.366524 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-podinfo\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.366780 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.366556 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-workload-certs\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.366780 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.366572 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8849f8ab-2b78-439c-911c-323c8c21c8b3-istiod-ca-cert\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.366780 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.366592 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-data\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.366780 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.366738 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-token\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.366780 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.366776 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-workload-socket\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.367106 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.366878 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-credential-socket\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.367106 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.366915 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-workload-certs\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.367106 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.367069 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-workload-socket\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.367288 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.367169 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-data\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.367660 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.367634 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8849f8ab-2b78-439c-911c-323c8c21c8b3-istiod-ca-cert\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.369127 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.369075 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-envoy\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.369222 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.369190 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-podinfo\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.391715 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.391684 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qk4s\" (UniqueName: \"kubernetes.io/projected/8849f8ab-2b78-439c-911c-323c8c21c8b3-kube-api-access-8qk4s\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.407669 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.407638 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-token\") pod \"e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.685450 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.685364 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:47.997109 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:47.996905 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw"] Sep 11 14:19:47.997340 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:19:47.997294 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8849f8ab_2b78_439c_911c_323c8c21c8b3.slice/crio-4447723e9abf853b7739080134c71e4a7841d2acf91996af9ed744c5bc23138b WatchSource:0}: Error finding container 4447723e9abf853b7739080134c71e4a7841d2acf91996af9ed744c5bc23138b: Status 404 returned error can't find the container with id 4447723e9abf853b7739080134c71e4a7841d2acf91996af9ed744c5bc23138b Sep 11 14:19:48.000472 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:48.000430 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:19:48.000576 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:48.000522 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:19:48.000576 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:48.000561 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:19:48.728728 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:48.728685 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" event={"ID":"8849f8ab-2b78-439c-911c-323c8c21c8b3","Type":"ContainerStarted","Data":"4d7047beb7446609439de1e9b469e39aecd1a293a92854a7c725a86bfcf138dd"} Sep 11 14:19:48.729176 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:48.728755 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" event={"ID":"8849f8ab-2b78-439c-911c-323c8c21c8b3","Type":"ContainerStarted","Data":"4447723e9abf853b7739080134c71e4a7841d2acf91996af9ed744c5bc23138b"} Sep 11 14:19:48.767714 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:48.767643 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" podStartSLOduration=1.767621731 podStartE2EDuration="1.767621731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:48.765142504 +0000 UTC m=+1329.170123629" watchObservedRunningTime="2026-09-11 14:19:48.767621731 +0000 UTC m=+1329.172602874" Sep 11 14:19:49.666282 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:49.666242 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 14:19:49.666465 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:49.666320 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" podUID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 14:19:49.685575 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:49.685540 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:49.690746 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:49.690716 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:49.736037 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:49.736007 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:49.737223 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:49.737199 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:19:50.744366 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:50.744329 2566 generic.go:356] "Generic (PLEG): container finished" podID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerID="a26be4307612c0e28d6434346cae6b5f33ecb22a415a72d3fb134c8a0588c1fe" exitCode=0 Sep 11 14:19:50.744814 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:50.744400 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" event={"ID":"a5f713d3-d284-4778-b474-1df1a1d06d17","Type":"ContainerDied","Data":"a26be4307612c0e28d6434346cae6b5f33ecb22a415a72d3fb134c8a0588c1fe"} Sep 11 14:19:50.744814 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:50.744778 2566 scope.go:117] "RemoveContainer" containerID="a26be4307612c0e28d6434346cae6b5f33ecb22a415a72d3fb134c8a0588c1fe" Sep 11 14:19:50.745799 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:50.745772 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:19:50.745921 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:50.745893 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:19:50.745988 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:50.745948 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:19:51.311724 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.311659 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:19:51.406246 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.406209 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-data\") pod \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " Sep 11 14:19:51.406448 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.406265 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istiod-ca-cert\") pod \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " Sep 11 14:19:51.406448 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.406300 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-workload-certs\") pod \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " Sep 11 14:19:51.406448 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.406322 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-credential-socket\") pod \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " Sep 11 14:19:51.406448 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.406345 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-token\") pod \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " Sep 11 14:19:51.406448 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.406396 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-envoy\") pod \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " Sep 11 14:19:51.406690 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.406458 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmpwm\" (UniqueName: \"kubernetes.io/projected/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-kube-api-access-zmpwm\") pod \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " Sep 11 14:19:51.406690 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.406524 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-podinfo\") pod \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " Sep 11 14:19:51.406690 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.406562 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" (UID: "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:51.406690 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.406588 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" (UID: "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:51.406690 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.406663 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-workload-socket\") pod \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\" (UID: \"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3\") " Sep 11 14:19:51.406690 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.406674 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" (UID: "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:51.406999 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.406956 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" (UID: "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:19:51.406999 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.406963 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" (UID: "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:51.407130 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.407055 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:51.407130 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.407076 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:51.407130 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.407110 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:51.407130 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.407122 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:51.407282 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.407135 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:51.408854 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.408827 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" (UID: "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:19:51.408854 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.408831 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-kube-api-access-zmpwm" (OuterVolumeSpecName: "kube-api-access-zmpwm") pod "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" (UID: "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3"). InnerVolumeSpecName "kube-api-access-zmpwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:51.409284 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.409265 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" (UID: "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:51.409344 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.409298 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" (UID: "5f0878d1-48b2-4cd2-865b-c6f21e95a0a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:51.508624 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.508585 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:51.508624 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.508620 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:51.508884 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.508634 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmpwm\" (UniqueName: \"kubernetes.io/projected/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-kube-api-access-zmpwm\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:51.508884 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.508647 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:19:51.753496 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.753453 2566 generic.go:356] "Generic (PLEG): container finished" podID="5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" containerID="f8373d228c54125a4ed5d1dfa09f501dbeece8f3f5e3840b0bc487bba2f3aebf" exitCode=0 Sep 11 14:19:51.753956 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.753505 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" Sep 11 14:19:51.753956 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.753556 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" event={"ID":"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3","Type":"ContainerDied","Data":"f8373d228c54125a4ed5d1dfa09f501dbeece8f3f5e3840b0bc487bba2f3aebf"} Sep 11 14:19:51.753956 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.753617 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55" event={"ID":"5f0878d1-48b2-4cd2-865b-c6f21e95a0a3","Type":"ContainerDied","Data":"15f2593f5a27c30cea5f284efff74c7e01b2fdbbb8e922caf8e6b6bc6af80ac6"} Sep 11 14:19:51.753956 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.753641 2566 scope.go:117] "RemoveContainer" containerID="f8373d228c54125a4ed5d1dfa09f501dbeece8f3f5e3840b0bc487bba2f3aebf" Sep 11 14:19:51.755831 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.755804 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" event={"ID":"a5f713d3-d284-4778-b474-1df1a1d06d17","Type":"ContainerStarted","Data":"0b9c9cb7efa8438ad6a24ade743d3ca39789ebbb5e4b45a396df5455cc86ba64"} Sep 11 14:19:51.764837 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.764818 2566 scope.go:117] "RemoveContainer" containerID="d20182eacb3efcc536b185de20c41ef5cc9d8d7062af208380b975e04ef9c8c6" Sep 11 14:19:51.774205 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.774182 2566 scope.go:117] "RemoveContainer" containerID="f8373d228c54125a4ed5d1dfa09f501dbeece8f3f5e3840b0bc487bba2f3aebf" Sep 11 14:19:51.774521 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:19:51.774502 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8373d228c54125a4ed5d1dfa09f501dbeece8f3f5e3840b0bc487bba2f3aebf\": container with ID starting with f8373d228c54125a4ed5d1dfa09f501dbeece8f3f5e3840b0bc487bba2f3aebf not found: ID does not exist" containerID="f8373d228c54125a4ed5d1dfa09f501dbeece8f3f5e3840b0bc487bba2f3aebf" Sep 11 14:19:51.774586 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.774531 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8373d228c54125a4ed5d1dfa09f501dbeece8f3f5e3840b0bc487bba2f3aebf"} err="failed to get container status \"f8373d228c54125a4ed5d1dfa09f501dbeece8f3f5e3840b0bc487bba2f3aebf\": rpc error: code = NotFound desc = could not find container \"f8373d228c54125a4ed5d1dfa09f501dbeece8f3f5e3840b0bc487bba2f3aebf\": container with ID starting with f8373d228c54125a4ed5d1dfa09f501dbeece8f3f5e3840b0bc487bba2f3aebf not found: ID does not exist" Sep 11 14:19:51.774586 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.774549 2566 scope.go:117] "RemoveContainer" containerID="d20182eacb3efcc536b185de20c41ef5cc9d8d7062af208380b975e04ef9c8c6" Sep 11 14:19:51.774834 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:19:51.774814 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d20182eacb3efcc536b185de20c41ef5cc9d8d7062af208380b975e04ef9c8c6\": container with ID starting with d20182eacb3efcc536b185de20c41ef5cc9d8d7062af208380b975e04ef9c8c6 not found: ID does not exist" containerID="d20182eacb3efcc536b185de20c41ef5cc9d8d7062af208380b975e04ef9c8c6" Sep 11 14:19:51.774882 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.774840 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d20182eacb3efcc536b185de20c41ef5cc9d8d7062af208380b975e04ef9c8c6"} err="failed to get container status \"d20182eacb3efcc536b185de20c41ef5cc9d8d7062af208380b975e04ef9c8c6\": rpc error: code = NotFound desc = could not find container \"d20182eacb3efcc536b185de20c41ef5cc9d8d7062af208380b975e04ef9c8c6\": container with ID starting with d20182eacb3efcc536b185de20c41ef5cc9d8d7062af208380b975e04ef9c8c6 not found: ID does not exist" Sep 11 14:19:51.809108 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.808199 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55"] Sep 11 14:19:51.812218 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.812184 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c6156e-openshift-default-67dd98c676-dbv55"] Sep 11 14:19:51.954537 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.954496 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:51.954864 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.954839 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Sep 11 14:19:51.954964 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:51.954912 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" podUID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Sep 11 14:19:52.097869 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:52.097765 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" path="/var/lib/kubelet/pods/5f0878d1-48b2-4cd2-865b-c6f21e95a0a3/volumes" Sep 11 14:19:52.954336 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:52.954298 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Sep 11 14:19:52.954857 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:52.954398 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" podUID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Sep 11 14:19:53.954936 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:53.954893 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Sep 11 14:19:53.955367 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:53.954972 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" podUID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Sep 11 14:19:54.495012 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.494977 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz"] Sep 11 14:19:54.495427 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.495414 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" containerName="istio-proxy" Sep 11 14:19:54.495486 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.495429 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fffb9f82-116a-4c91-bb26-3fd11261e4b8" containerName="istio-proxy" Sep 11 14:19:54.495486 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.495454 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" containerName="istio-proxy" Sep 11 14:19:54.495486 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.495459 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" containerName="istio-proxy" Sep 11 14:19:54.495486 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.495477 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" containerName="istio-proxy" Sep 11 14:19:54.495486 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.495483 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" containerName="istio-proxy" Sep 11 14:19:54.495655 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.495545 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" containerName="istio-proxy" Sep 11 14:19:54.495655 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.495558 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f0878d1-48b2-4cd2-865b-c6f21e95a0a3" containerName="istio-proxy" Sep 11 14:19:54.502413 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.502376 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.515719 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.515657 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-335a68-openshift-default-dockercfg-5xk5r\"" Sep 11 14:19:54.535803 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.535772 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz"] Sep 11 14:19:54.540599 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.540570 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-envoy\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.540762 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.540607 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-data\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.540762 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.540648 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-token\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.540762 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.540676 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-credential-socket\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.540762 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.540709 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-workload-certs\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.540762 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.540735 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-podinfo\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.540762 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.540756 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-workload-socket\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.541125 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.540803 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7djh\" (UniqueName: \"kubernetes.io/projected/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-kube-api-access-r7djh\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.541125 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.540841 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istiod-ca-cert\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.642070 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.642001 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-envoy\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.642070 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.642046 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-data\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.642398 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.642117 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-token\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.642398 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.642147 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-credential-socket\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.642398 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.642178 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-workload-certs\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.642398 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.642208 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-podinfo\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.642398 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.642259 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-workload-socket\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.642398 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.642319 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7djh\" (UniqueName: \"kubernetes.io/projected/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-kube-api-access-r7djh\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.642398 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.642355 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istiod-ca-cert\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.642797 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.642586 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-data\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.642797 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.642714 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-credential-socket\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.642916 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.642803 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-workload-socket\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.643075 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.643050 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istiod-ca-cert\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.643386 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.643365 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-workload-certs\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.645460 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.645425 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-podinfo\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.645609 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.645590 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-envoy\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.659363 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.659340 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7djh\" (UniqueName: \"kubernetes.io/projected/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-kube-api-access-r7djh\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.661131 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.661109 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-token\") pod \"e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.822005 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.821904 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:54.954172 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.954132 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Sep 11 14:19:54.954339 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:54.954194 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" podUID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Sep 11 14:19:55.117884 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:55.116604 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz"] Sep 11 14:19:55.121621 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:19:55.120762 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ad9660_2319_4d8f_a3f7_d3a61e50d6dd.slice/crio-3eb8ba65ce8828fd53a65ecab38828839072f063cf4af94a4d07f75d482680b1 WatchSource:0}: Error finding container 3eb8ba65ce8828fd53a65ecab38828839072f063cf4af94a4d07f75d482680b1: Status 404 returned error can't find the container with id 3eb8ba65ce8828fd53a65ecab38828839072f063cf4af94a4d07f75d482680b1 Sep 11 14:19:55.127145 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:55.124183 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:19:55.127145 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:55.124269 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:19:55.127145 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:55.124313 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:19:55.785495 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:55.785463 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" event={"ID":"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd","Type":"ContainerStarted","Data":"024f45de152d457f7913936ac60e306f3af70bf9dd5ef5e7b7a9b38763d76d9c"} Sep 11 14:19:55.785683 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:55.785512 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" event={"ID":"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd","Type":"ContainerStarted","Data":"3eb8ba65ce8828fd53a65ecab38828839072f063cf4af94a4d07f75d482680b1"} Sep 11 14:19:55.822584 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:55.822550 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:55.954488 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:55.954446 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Sep 11 14:19:55.954701 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:55.954507 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" podUID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Sep 11 14:19:56.828718 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:56.828687 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:56.858734 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:56.858676 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" podStartSLOduration=2.858658957 podStartE2EDuration="2.858658957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:55.823067422 +0000 UTC m=+1336.228048551" watchObservedRunningTime="2026-09-11 14:19:56.858658957 +0000 UTC m=+1337.263640081" Sep 11 14:19:56.953951 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:56.953920 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Sep 11 14:19:56.954141 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:56.953973 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" podUID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Sep 11 14:19:57.798513 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:57.798478 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:57.799778 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:57.799751 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:19:57.957851 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:57.957819 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:57.958419 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:57.958331 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:19:57.959273 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:19:57.959251 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:20:05.777285 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:05.777236 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 14:20:05.777762 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:05.777311 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 14:20:06.329681 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.329649 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5"] Sep 11 14:20:06.333060 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.333036 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.336929 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.336901 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-031a56-openshift-default-dockercfg-4gxjj\"" Sep 11 14:20:06.425641 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.425601 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5"] Sep 11 14:20:06.475257 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.475213 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-istio-envoy\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.475257 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.475260 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-workload-certs\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.475483 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.475294 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq5fp\" (UniqueName: \"kubernetes.io/projected/adb57f87-d90c-421c-9751-70026534903d-kube-api-access-gq5fp\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.475483 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.475381 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb57f87-d90c-421c-9751-70026534903d-istiod-ca-cert\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.475483 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.475405 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-workload-socket\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.475483 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.475423 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb57f87-d90c-421c-9751-70026534903d-istio-token\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.475641 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.475472 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-credential-socket\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.475641 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.475513 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb57f87-d90c-421c-9751-70026534903d-istio-podinfo\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.475641 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.475562 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-istio-data\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.576747 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.576711 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-istio-data\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.576898 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.576811 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-istio-envoy\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.576898 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.576866 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-workload-certs\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.577012 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.576928 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq5fp\" (UniqueName: \"kubernetes.io/projected/adb57f87-d90c-421c-9751-70026534903d-kube-api-access-gq5fp\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.577065 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.577017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb57f87-d90c-421c-9751-70026534903d-istiod-ca-cert\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.577065 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.577044 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-workload-socket\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.577195 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.577077 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb57f87-d90c-421c-9751-70026534903d-istio-token\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.577195 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.577128 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-credential-socket\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.577195 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.577152 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb57f87-d90c-421c-9751-70026534903d-istio-podinfo\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.577341 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.577192 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-istio-data\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.577341 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.577280 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-workload-certs\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.577565 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.577544 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-credential-socket\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.577619 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.577561 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-workload-socket\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.577721 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.577703 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb57f87-d90c-421c-9751-70026534903d-istiod-ca-cert\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.579456 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.579434 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-istio-envoy\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.579896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.579846 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb57f87-d90c-421c-9751-70026534903d-istio-podinfo\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.603807 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.603774 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq5fp\" (UniqueName: \"kubernetes.io/projected/adb57f87-d90c-421c-9751-70026534903d-kube-api-access-gq5fp\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.604375 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.604347 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb57f87-d90c-421c-9751-70026534903d-istio-token\") pod \"e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.645496 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.645454 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:06.974531 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:20:06.974504 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb57f87_d90c_421c_9751_70026534903d.slice/crio-6a3fe3b455da864101d939b036dfb55e221f4db804bdda747d67d6555cdac5a8 WatchSource:0}: Error finding container 6a3fe3b455da864101d939b036dfb55e221f4db804bdda747d67d6555cdac5a8: Status 404 returned error can't find the container with id 6a3fe3b455da864101d939b036dfb55e221f4db804bdda747d67d6555cdac5a8 Sep 11 14:20:06.976146 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.976122 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5"] Sep 11 14:20:06.977419 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.977382 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:06.977488 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.977471 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:06.977535 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:06.977520 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:07.880894 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:07.879498 2566 generic.go:356] "Generic (PLEG): container finished" podID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerID="4d7047beb7446609439de1e9b469e39aecd1a293a92854a7c725a86bfcf138dd" exitCode=0 Sep 11 14:20:07.880894 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:07.879679 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" event={"ID":"8849f8ab-2b78-439c-911c-323c8c21c8b3","Type":"ContainerDied","Data":"4d7047beb7446609439de1e9b469e39aecd1a293a92854a7c725a86bfcf138dd"} Sep 11 14:20:07.880894 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:07.880257 2566 scope.go:117] "RemoveContainer" containerID="4d7047beb7446609439de1e9b469e39aecd1a293a92854a7c725a86bfcf138dd" Sep 11 14:20:07.883143 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:07.882745 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:07.883143 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:07.882835 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:07.883143 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:07.882907 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:07.901754 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:07.899630 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" event={"ID":"adb57f87-d90c-421c-9751-70026534903d","Type":"ContainerStarted","Data":"cf4c0aeee0043b2eea346a3667cd18246b4f4f11cc5309d9c3900fa4cac91eb8"} Sep 11 14:20:07.901754 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:07.899696 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" event={"ID":"adb57f87-d90c-421c-9751-70026534903d","Type":"ContainerStarted","Data":"6a3fe3b455da864101d939b036dfb55e221f4db804bdda747d67d6555cdac5a8"} Sep 11 14:20:07.998114 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:07.996770 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" podStartSLOduration=1.9967476259999999 podStartE2EDuration="1.996747626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:07.991328386 +0000 UTC m=+1348.396309515" watchObservedRunningTime="2026-09-11 14:20:07.996747626 +0000 UTC m=+1348.401728756" Sep 11 14:20:08.645706 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:08.645667 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:08.650838 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:08.650810 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:08.920519 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:08.920417 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" event={"ID":"8849f8ab-2b78-439c-911c-323c8c21c8b3","Type":"ContainerStarted","Data":"30c1f82f596d356199794ba8bd99e298dc2ad4ba0f2fe2a9db5fde11dac75044"} Sep 11 14:20:08.920519 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:08.920509 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:08.924585 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:08.924551 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:09.686022 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:09.685989 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:20:09.686465 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:09.686157 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 11 14:20:09.686465 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:09.686200 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 11 14:20:10.686458 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:10.686426 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 11 14:20:10.686851 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:10.686487 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 11 14:20:11.686045 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:11.686005 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 11 14:20:11.686250 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:11.686079 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 11 14:20:12.685781 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:12.685730 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 11 14:20:12.686217 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:12.685811 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 11 14:20:13.686754 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:13.686713 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 11 14:20:13.687181 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:13.686777 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 11 14:20:14.685735 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:14.685701 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 11 14:20:14.685974 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:14.685760 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 11 14:20:15.686573 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:15.686534 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Sep 11 14:20:15.686983 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:15.686604 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Sep 11 14:20:16.690352 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:16.690323 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:20:16.690738 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:16.690680 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:20:16.691454 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:16.691433 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:20:19.004508 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:19.004476 2566 generic.go:356] "Generic (PLEG): container finished" podID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" containerID="024f45de152d457f7913936ac60e306f3af70bf9dd5ef5e7b7a9b38763d76d9c" exitCode=0 Sep 11 14:20:19.004985 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:19.004570 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" event={"ID":"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd","Type":"ContainerDied","Data":"024f45de152d457f7913936ac60e306f3af70bf9dd5ef5e7b7a9b38763d76d9c"} Sep 11 14:20:19.005160 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:19.005144 2566 scope.go:117] "RemoveContainer" containerID="024f45de152d457f7913936ac60e306f3af70bf9dd5ef5e7b7a9b38763d76d9c" Sep 11 14:20:19.006309 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:19.006280 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:19.006412 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:19.006353 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:19.006412 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:19.006386 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:20.013777 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:20.013744 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" event={"ID":"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd","Type":"ContainerStarted","Data":"a4cc6298c85ce6bad4c3d54707e5c8237da0c27374f9dde45f34b6675772e940"} Sep 11 14:20:20.823008 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:20.822966 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:20:20.823300 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:20.823271 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Sep 11 14:20:20.823392 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:20.823330 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" podUID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Sep 11 14:20:21.822636 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:21.822592 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Sep 11 14:20:21.823055 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:21.822679 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" podUID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Sep 11 14:20:22.823452 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:22.823412 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Sep 11 14:20:22.823873 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:22.823482 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" podUID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Sep 11 14:20:23.330882 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.330842 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr"] Sep 11 14:20:23.334418 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.334389 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.339593 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.339567 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-557930d3-openshift-default-dockercfg-v66cj\"" Sep 11 14:20:23.356144 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.356115 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr"] Sep 11 14:20:23.379312 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.379277 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrbr6\" (UniqueName: \"kubernetes.io/projected/b281c7b0-b3c7-487b-9502-eb73929fb2ba-kube-api-access-hrbr6\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.379487 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.379342 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-workload-socket\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.379487 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.379381 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-workload-certs\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.379487 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.379426 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-token\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.379487 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.379449 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istiod-ca-cert\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.379670 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.379534 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-credential-socket\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.379670 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.379609 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-podinfo\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.379670 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.379643 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-data\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.379784 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.379674 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-envoy\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.482578 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.481413 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-credential-socket\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.482578 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.481688 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-podinfo\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.482578 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.481801 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-data\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.482578 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.481840 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-envoy\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.482578 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.481887 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-credential-socket\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.482578 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.481941 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrbr6\" (UniqueName: \"kubernetes.io/projected/b281c7b0-b3c7-487b-9502-eb73929fb2ba-kube-api-access-hrbr6\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.482578 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.481985 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-workload-socket\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.482578 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.482022 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-workload-certs\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.482578 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.482054 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-token\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.482578 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.482078 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istiod-ca-cert\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.483287 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.482758 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-workload-certs\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.483287 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.482871 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istiod-ca-cert\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.483287 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.483046 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-workload-socket\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.483287 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.483215 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-data\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.487625 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.485681 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-envoy\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.487625 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.486576 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-podinfo\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.500988 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.500929 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrbr6\" (UniqueName: \"kubernetes.io/projected/b281c7b0-b3c7-487b-9502-eb73929fb2ba-kube-api-access-hrbr6\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.502057 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.501539 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-token\") pod \"e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.646052 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.645962 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:23.824022 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.823547 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Sep 11 14:20:23.824022 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.823617 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" podUID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Sep 11 14:20:23.970966 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.970940 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr"] Sep 11 14:20:23.978909 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:20:23.978867 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb281c7b0_b3c7_487b_9502_eb73929fb2ba.slice/crio-f9ea1a6135f46205bb816988d41838b715759ddded51e34e2f2010c5568276da WatchSource:0}: Error finding container f9ea1a6135f46205bb816988d41838b715759ddded51e34e2f2010c5568276da: Status 404 returned error can't find the container with id f9ea1a6135f46205bb816988d41838b715759ddded51e34e2f2010c5568276da Sep 11 14:20:23.981584 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.981543 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:23.981811 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.981781 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:23.981942 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:23.981835 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:24.047699 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:24.047663 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" event={"ID":"b281c7b0-b3c7-487b-9502-eb73929fb2ba","Type":"ContainerStarted","Data":"f9ea1a6135f46205bb816988d41838b715759ddded51e34e2f2010c5568276da"} Sep 11 14:20:24.822900 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:24.822852 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Sep 11 14:20:24.823152 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:24.822924 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" podUID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Sep 11 14:20:25.060397 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:25.060353 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" event={"ID":"b281c7b0-b3c7-487b-9502-eb73929fb2ba","Type":"ContainerStarted","Data":"ba0756e6b11c5f43e69759071a4c4ae805a8b870051f0db26a63bcceded476a7"} Sep 11 14:20:25.087766 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:25.087661 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" podStartSLOduration=2.0876447 podStartE2EDuration="2.0876447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:20:25.084068288 +0000 UTC m=+1365.489049415" watchObservedRunningTime="2026-09-11 14:20:25.0876447 +0000 UTC m=+1365.492625828" Sep 11 14:20:25.646461 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:25.646424 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:25.651404 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:25.651377 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:25.866488 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:25.866427 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:20:25.866816 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:25.866779 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:20:25.867955 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:25.867938 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:20:26.066807 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:26.066776 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:26.067783 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:26.067760 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:34.123203 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:34.123167 2566 generic.go:356] "Generic (PLEG): container finished" podID="adb57f87-d90c-421c-9751-70026534903d" containerID="cf4c0aeee0043b2eea346a3667cd18246b4f4f11cc5309d9c3900fa4cac91eb8" exitCode=0 Sep 11 14:20:34.123639 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:34.123237 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" event={"ID":"adb57f87-d90c-421c-9751-70026534903d","Type":"ContainerDied","Data":"cf4c0aeee0043b2eea346a3667cd18246b4f4f11cc5309d9c3900fa4cac91eb8"} Sep 11 14:20:34.123732 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:34.123714 2566 scope.go:117] "RemoveContainer" containerID="cf4c0aeee0043b2eea346a3667cd18246b4f4f11cc5309d9c3900fa4cac91eb8" Sep 11 14:20:34.124881 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:34.124844 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:34.124980 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:34.124935 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:34.124980 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:34.124965 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:35.133193 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:35.133154 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" event={"ID":"adb57f87-d90c-421c-9751-70026534903d","Type":"ContainerStarted","Data":"8d7ef5a349b20b9da7f75c7a841c0b545925d4708f5117cda8789f47661a5af7"} Sep 11 14:20:35.646078 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:35.646040 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:35.647336 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:35.647309 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 11 14:20:35.647462 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:35.647356 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" podUID="adb57f87-d90c-421c-9751-70026534903d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 11 14:20:36.645990 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:36.645948 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 11 14:20:36.646509 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:36.646025 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" podUID="adb57f87-d90c-421c-9751-70026534903d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 11 14:20:37.646020 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:37.645978 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 11 14:20:37.646455 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:37.646043 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" podUID="adb57f87-d90c-421c-9751-70026534903d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 11 14:20:38.645853 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:38.645820 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 11 14:20:38.646066 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:38.645876 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" podUID="adb57f87-d90c-421c-9751-70026534903d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 11 14:20:38.920500 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:38.920403 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:39.646479 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:39.646441 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Sep 11 14:20:39.646881 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:39.646500 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" podUID="adb57f87-d90c-421c-9751-70026534903d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Sep 11 14:20:40.281928 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:40.281890 2566 scope.go:117] "RemoveContainer" containerID="d9cd362d4d85b62f5e2eea00f2bbab4c5c68c14086881d0f0b0c87ccc220a718" Sep 11 14:20:40.303592 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:40.303561 2566 scope.go:117] "RemoveContainer" containerID="70c2d47a495db435fc71a869933b1c83c12ebd7e1f9e3ed4969b728b6ba9dcb8" Sep 11 14:20:40.650562 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:40.650424 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:40.651591 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:40.651570 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:20:51.246381 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:51.246340 2566 generic.go:356] "Generic (PLEG): container finished" podID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" containerID="ba0756e6b11c5f43e69759071a4c4ae805a8b870051f0db26a63bcceded476a7" exitCode=0 Sep 11 14:20:51.246812 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:51.246414 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" event={"ID":"b281c7b0-b3c7-487b-9502-eb73929fb2ba","Type":"ContainerDied","Data":"ba0756e6b11c5f43e69759071a4c4ae805a8b870051f0db26a63bcceded476a7"} Sep 11 14:20:51.246882 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:51.246866 2566 scope.go:117] "RemoveContainer" containerID="ba0756e6b11c5f43e69759071a4c4ae805a8b870051f0db26a63bcceded476a7" Sep 11 14:20:51.248195 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:51.248171 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:51.248277 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:51.248236 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:51.248277 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:51.248266 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:52.254553 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:52.254517 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" event={"ID":"b281c7b0-b3c7-487b-9502-eb73929fb2ba","Type":"ContainerStarted","Data":"2e3160186e1192c9d19ab1c2b1b229ec34efbaf54b86aa969678108f69aa4239"} Sep 11 14:20:52.646981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:52.646889 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:52.648203 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:52.648183 2566 patch_prober.go:28] interesting pod/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Sep 11 14:20:52.648265 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:52.648246 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" podUID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Sep 11 14:20:53.646504 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:53.646465 2566 patch_prober.go:28] interesting pod/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Sep 11 14:20:53.646912 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:53.646534 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" podUID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Sep 11 14:20:54.646754 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:54.646718 2566 patch_prober.go:28] interesting pod/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Sep 11 14:20:54.647182 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:54.646785 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" podUID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Sep 11 14:20:55.647010 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:55.646975 2566 patch_prober.go:28] interesting pod/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Sep 11 14:20:55.647442 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:55.647066 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" podUID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Sep 11 14:20:56.067675 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:56.067637 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:56.646758 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:56.646725 2566 patch_prober.go:28] interesting pod/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Sep 11 14:20:56.646917 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:56.646778 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" podUID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Sep 11 14:20:57.652181 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:57.652149 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:20:57.653187 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:20:57.653164 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:21:40.342793 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:21:40.342715 2566 scope.go:117] "RemoveContainer" containerID="15220d652f552f9c23f2a8b3ac22b781679169a593612506ed42bef2a0caaa82" Sep 11 14:21:40.352287 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:21:40.352265 2566 scope.go:117] "RemoveContainer" containerID="7a56c43d3fc420ca52309f7c542e4bc536167560dd48cd2f5c047289e15895c7" Sep 11 14:21:40.361272 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:21:40.361247 2566 scope.go:117] "RemoveContainer" containerID="0cecf60879aaa7ec3951a54717f6de04eb17a8a972de52244e28a4d7ca053f25" Sep 11 14:21:40.369782 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:21:40.369763 2566 scope.go:117] "RemoveContainer" containerID="087ee3c0d89494a29421de1c03e00db3672f9a49d3f9ad109d5ee3b29e94c062" Sep 11 14:21:58.878524 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:21:58.878493 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557930d3-w78j2"] Sep 11 14:21:58.889049 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:21:58.889012 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557930d3-w78j2" Sep 11 14:21:58.891292 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:21:58.891249 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557930d3-w78j2"] Sep 11 14:21:58.892439 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:21:58.892416 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:21:58.996382 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:21:58.996347 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55cb2d8a-0906-4c60-bc50-3dcda66cf042-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-557930d3-w78j2\" (UID: \"55cb2d8a-0906-4c60-bc50-3dcda66cf042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557930d3-w78j2" Sep 11 14:21:59.097288 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:21:59.097254 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55cb2d8a-0906-4c60-bc50-3dcda66cf042-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-557930d3-w78j2\" (UID: \"55cb2d8a-0906-4c60-bc50-3dcda66cf042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557930d3-w78j2" Sep 11 14:21:59.097914 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:21:59.097894 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55cb2d8a-0906-4c60-bc50-3dcda66cf042-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-557930d3-w78j2\" (UID: \"55cb2d8a-0906-4c60-bc50-3dcda66cf042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557930d3-w78j2" Sep 11 14:21:59.202612 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:21:59.202520 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557930d3-w78j2" Sep 11 14:21:59.481508 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:21:59.481479 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557930d3-w78j2"] Sep 11 14:21:59.482249 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:21:59.482225 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55cb2d8a_0906_4c60_bc50_3dcda66cf042.slice/crio-2f523b90ee4626acdd1d40dfa355f3d66291708192a82e9c344a9ba91b87c102 WatchSource:0}: Error finding container 2f523b90ee4626acdd1d40dfa355f3d66291708192a82e9c344a9ba91b87c102: Status 404 returned error can't find the container with id 2f523b90ee4626acdd1d40dfa355f3d66291708192a82e9c344a9ba91b87c102 Sep 11 14:21:59.709539 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:21:59.709504 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557930d3-w78j2" event={"ID":"55cb2d8a-0906-4c60-bc50-3dcda66cf042","Type":"ContainerStarted","Data":"2f523b90ee4626acdd1d40dfa355f3d66291708192a82e9c344a9ba91b87c102"} Sep 11 14:22:00.718033 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:00.717939 2566 generic.go:356] "Generic (PLEG): container finished" podID="55cb2d8a-0906-4c60-bc50-3dcda66cf042" containerID="7ef1ce14d6ddb92a72649cc2aa5950d37f36e78b0d3ed8737fe5f4ac839283d9" exitCode=0 Sep 11 14:22:00.718447 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:00.718025 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557930d3-w78j2" event={"ID":"55cb2d8a-0906-4c60-bc50-3dcda66cf042","Type":"ContainerDied","Data":"7ef1ce14d6ddb92a72649cc2aa5950d37f36e78b0d3ed8737fe5f4ac839283d9"} Sep 11 14:22:01.867027 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:01.867004 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557930d3-w78j2" Sep 11 14:22:01.923456 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:01.923428 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55cb2d8a-0906-4c60-bc50-3dcda66cf042-maas-api-service-ca\") pod \"55cb2d8a-0906-4c60-bc50-3dcda66cf042\" (UID: \"55cb2d8a-0906-4c60-bc50-3dcda66cf042\") " Sep 11 14:22:01.923749 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:01.923728 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55cb2d8a-0906-4c60-bc50-3dcda66cf042-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55cb2d8a-0906-4c60-bc50-3dcda66cf042" (UID: "55cb2d8a-0906-4c60-bc50-3dcda66cf042"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:22:02.024818 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:02.024737 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55cb2d8a-0906-4c60-bc50-3dcda66cf042-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:22:02.733148 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:02.733120 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557930d3-w78j2" Sep 11 14:22:02.733319 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:02.733143 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557930d3-w78j2" event={"ID":"55cb2d8a-0906-4c60-bc50-3dcda66cf042","Type":"ContainerDied","Data":"2f523b90ee4626acdd1d40dfa355f3d66291708192a82e9c344a9ba91b87c102"} Sep 11 14:22:02.733319 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:02.733199 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f523b90ee4626acdd1d40dfa355f3d66291708192a82e9c344a9ba91b87c102" Sep 11 14:22:15.025448 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:15.025411 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557930d3-w78j2"] Sep 11 14:22:15.033226 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:15.033203 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557930d3-w78j2"] Sep 11 14:22:16.100251 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:16.100203 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55cb2d8a-0906-4c60-bc50-3dcda66cf042" path="/var/lib/kubelet/pods/55cb2d8a-0906-4c60-bc50-3dcda66cf042/volumes" Sep 11 14:22:40.197483 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:40.197453 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:22:40.198884 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:40.198863 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:22:40.405410 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:40.405379 2566 scope.go:117] "RemoveContainer" containerID="7eb966c1c1d6fb7a1608d37255acb8bc88fd6fa8e854d28701ef1bf14741b6d2" Sep 11 14:22:40.416994 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:22:40.416973 2566 scope.go:117] "RemoveContainer" containerID="478d354fec8a2376894b376acd1204eb3ca23dcc9bfd895049fd4cdfea490bff" Sep 11 14:23:08.606675 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:08.606581 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr"] Sep 11 14:23:08.607306 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:08.606815 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" podUID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" containerName="istio-proxy" containerID="cri-o://2e3160186e1192c9d19ab1c2b1b229ec34efbaf54b86aa969678108f69aa4239" gracePeriod=30 Sep 11 14:23:12.655108 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:12.655046 2566 patch_prober.go:28] interesting pod/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:23:12.655514 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:12.655156 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" podUID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:23:13.866707 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.866642 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:23:13.952259 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952226 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istiod-ca-cert\") pod \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " Sep 11 14:23:13.952436 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952285 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrbr6\" (UniqueName: \"kubernetes.io/projected/b281c7b0-b3c7-487b-9502-eb73929fb2ba-kube-api-access-hrbr6\") pod \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " Sep 11 14:23:13.952436 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952313 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-workload-certs\") pod \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " Sep 11 14:23:13.952436 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952355 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-workload-socket\") pod \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " Sep 11 14:23:13.952436 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952370 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-credential-socket\") pod \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " Sep 11 14:23:13.952436 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952387 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-token\") pod \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " Sep 11 14:23:13.952436 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952407 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-data\") pod \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " Sep 11 14:23:13.952753 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952488 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-envoy\") pod \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " Sep 11 14:23:13.952753 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952508 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-podinfo\") pod \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\" (UID: \"b281c7b0-b3c7-487b-9502-eb73929fb2ba\") " Sep 11 14:23:13.952753 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952688 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b281c7b0-b3c7-487b-9502-eb73929fb2ba" (UID: "b281c7b0-b3c7-487b-9502-eb73929fb2ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:13.952753 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952698 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b281c7b0-b3c7-487b-9502-eb73929fb2ba" (UID: "b281c7b0-b3c7-487b-9502-eb73929fb2ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:13.952753 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952710 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b281c7b0-b3c7-487b-9502-eb73929fb2ba" (UID: "b281c7b0-b3c7-487b-9502-eb73929fb2ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:23:13.952753 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952720 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b281c7b0-b3c7-487b-9502-eb73929fb2ba" (UID: "b281c7b0-b3c7-487b-9502-eb73929fb2ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:13.953102 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952857 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:23:13.953102 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952871 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:23:13.953102 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952890 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:23:13.953102 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952899 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:23:13.953102 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.952927 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-data" (OuterVolumeSpecName: "istio-data") pod "b281c7b0-b3c7-487b-9502-eb73929fb2ba" (UID: "b281c7b0-b3c7-487b-9502-eb73929fb2ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:13.955109 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.955046 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-token" (OuterVolumeSpecName: "istio-token") pod "b281c7b0-b3c7-487b-9502-eb73929fb2ba" (UID: "b281c7b0-b3c7-487b-9502-eb73929fb2ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:13.955585 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.955269 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b281c7b0-b3c7-487b-9502-eb73929fb2ba-kube-api-access-hrbr6" (OuterVolumeSpecName: "kube-api-access-hrbr6") pod "b281c7b0-b3c7-487b-9502-eb73929fb2ba" (UID: "b281c7b0-b3c7-487b-9502-eb73929fb2ba"). InnerVolumeSpecName "kube-api-access-hrbr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:13.955585 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.955307 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b281c7b0-b3c7-487b-9502-eb73929fb2ba" (UID: "b281c7b0-b3c7-487b-9502-eb73929fb2ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:23:13.955773 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:13.955750 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b281c7b0-b3c7-487b-9502-eb73929fb2ba" (UID: "b281c7b0-b3c7-487b-9502-eb73929fb2ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:14.053922 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.053888 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:23:14.053922 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.053915 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:23:14.053922 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.053928 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrbr6\" (UniqueName: \"kubernetes.io/projected/b281c7b0-b3c7-487b-9502-eb73929fb2ba-kube-api-access-hrbr6\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:23:14.054201 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.053937 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:23:14.054201 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.053945 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b281c7b0-b3c7-487b-9502-eb73929fb2ba-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:23:14.203224 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.203126 2566 generic.go:356] "Generic (PLEG): container finished" podID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" containerID="2e3160186e1192c9d19ab1c2b1b229ec34efbaf54b86aa969678108f69aa4239" exitCode=0 Sep 11 14:23:14.203224 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.203157 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" Sep 11 14:23:14.203224 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.203173 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" event={"ID":"b281c7b0-b3c7-487b-9502-eb73929fb2ba","Type":"ContainerDied","Data":"2e3160186e1192c9d19ab1c2b1b229ec34efbaf54b86aa969678108f69aa4239"} Sep 11 14:23:14.203521 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.203229 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr" event={"ID":"b281c7b0-b3c7-487b-9502-eb73929fb2ba","Type":"ContainerDied","Data":"f9ea1a6135f46205bb816988d41838b715759ddded51e34e2f2010c5568276da"} Sep 11 14:23:14.203521 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.203258 2566 scope.go:117] "RemoveContainer" containerID="2e3160186e1192c9d19ab1c2b1b229ec34efbaf54b86aa969678108f69aa4239" Sep 11 14:23:14.213011 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.212996 2566 scope.go:117] "RemoveContainer" containerID="ba0756e6b11c5f43e69759071a4c4ae805a8b870051f0db26a63bcceded476a7" Sep 11 14:23:14.221619 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.221601 2566 scope.go:117] "RemoveContainer" containerID="2e3160186e1192c9d19ab1c2b1b229ec34efbaf54b86aa969678108f69aa4239" Sep 11 14:23:14.221888 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:23:14.221869 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e3160186e1192c9d19ab1c2b1b229ec34efbaf54b86aa969678108f69aa4239\": container with ID starting with 2e3160186e1192c9d19ab1c2b1b229ec34efbaf54b86aa969678108f69aa4239 not found: ID does not exist" containerID="2e3160186e1192c9d19ab1c2b1b229ec34efbaf54b86aa969678108f69aa4239" Sep 11 14:23:14.221943 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.221896 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e3160186e1192c9d19ab1c2b1b229ec34efbaf54b86aa969678108f69aa4239"} err="failed to get container status \"2e3160186e1192c9d19ab1c2b1b229ec34efbaf54b86aa969678108f69aa4239\": rpc error: code = NotFound desc = could not find container \"2e3160186e1192c9d19ab1c2b1b229ec34efbaf54b86aa969678108f69aa4239\": container with ID starting with 2e3160186e1192c9d19ab1c2b1b229ec34efbaf54b86aa969678108f69aa4239 not found: ID does not exist" Sep 11 14:23:14.221943 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.221913 2566 scope.go:117] "RemoveContainer" containerID="ba0756e6b11c5f43e69759071a4c4ae805a8b870051f0db26a63bcceded476a7" Sep 11 14:23:14.222205 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:23:14.222184 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0756e6b11c5f43e69759071a4c4ae805a8b870051f0db26a63bcceded476a7\": container with ID starting with ba0756e6b11c5f43e69759071a4c4ae805a8b870051f0db26a63bcceded476a7 not found: ID does not exist" containerID="ba0756e6b11c5f43e69759071a4c4ae805a8b870051f0db26a63bcceded476a7" Sep 11 14:23:14.222266 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.222211 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0756e6b11c5f43e69759071a4c4ae805a8b870051f0db26a63bcceded476a7"} err="failed to get container status \"ba0756e6b11c5f43e69759071a4c4ae805a8b870051f0db26a63bcceded476a7\": rpc error: code = NotFound desc = could not find container \"ba0756e6b11c5f43e69759071a4c4ae805a8b870051f0db26a63bcceded476a7\": container with ID starting with ba0756e6b11c5f43e69759071a4c4ae805a8b870051f0db26a63bcceded476a7 not found: ID does not exist" Sep 11 14:23:14.233065 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.233045 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr"] Sep 11 14:23:14.241356 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:14.241338 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-557930d3-openshift-default-6d7c8b84dd-bd6tr"] Sep 11 14:23:16.099182 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:23:16.099133 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" path="/var/lib/kubelet/pods/b281c7b0-b3c7-487b-9502-eb73929fb2ba/volumes" Sep 11 14:24:06.301341 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.301303 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg"] Sep 11 14:24:06.301885 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.301864 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" containerName="istio-proxy" Sep 11 14:24:06.301885 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.301886 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" containerName="istio-proxy" Sep 11 14:24:06.302025 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.301897 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55cb2d8a-0906-4c60-bc50-3dcda66cf042" containerName="revoke-keys" Sep 11 14:24:06.302025 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.301907 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="55cb2d8a-0906-4c60-bc50-3dcda66cf042" containerName="revoke-keys" Sep 11 14:24:06.302025 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.301924 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" containerName="istio-proxy" Sep 11 14:24:06.302025 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.301933 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" containerName="istio-proxy" Sep 11 14:24:06.302242 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.302039 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="55cb2d8a-0906-4c60-bc50-3dcda66cf042" containerName="revoke-keys" Sep 11 14:24:06.302242 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.302051 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" containerName="istio-proxy" Sep 11 14:24:06.302242 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.302057 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b281c7b0-b3c7-487b-9502-eb73929fb2ba" containerName="istio-proxy" Sep 11 14:24:06.305954 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.305930 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.308574 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.308545 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d52edb-openshift-default-dockercfg-2v4p9\"" Sep 11 14:24:06.324860 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.324824 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg"] Sep 11 14:24:06.370353 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.370322 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istiod-ca-cert\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.370353 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.370356 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-workload-socket\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.370574 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.370384 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-podinfo\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.370574 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.370502 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-data\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.370574 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.370534 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-token\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.370688 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.370577 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-envoy\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.370688 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.370609 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zzpf\" (UniqueName: \"kubernetes.io/projected/c8cad07a-5d94-4e3f-ae91-5e8466250e78-kube-api-access-4zzpf\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.370688 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.370644 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-workload-certs\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.370688 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.370673 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-credential-socket\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.472759 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.472714 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-data\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.472988 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.472775 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-token\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.472988 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.472819 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-envoy\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.472988 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.472858 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zzpf\" (UniqueName: \"kubernetes.io/projected/c8cad07a-5d94-4e3f-ae91-5e8466250e78-kube-api-access-4zzpf\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.472988 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.472920 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-workload-certs\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.472988 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.472966 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-credential-socket\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.473396 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.473186 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istiod-ca-cert\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.473396 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.473225 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-workload-socket\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.473396 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.473276 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-podinfo\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.473566 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.473435 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-credential-socket\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.474181 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.474033 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-data\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.474601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.474573 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-workload-certs\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.474702 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.474583 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istiod-ca-cert\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.490112 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.474832 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-workload-socket\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.490112 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.477706 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-podinfo\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.490112 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.489293 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zzpf\" (UniqueName: \"kubernetes.io/projected/c8cad07a-5d94-4e3f-ae91-5e8466250e78-kube-api-access-4zzpf\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.492636 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.492332 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-envoy\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.492636 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.492604 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-token\") pod \"e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.617331 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.617247 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:06.911067 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.911032 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg"] Sep 11 14:24:06.912412 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:24:06.912382 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8cad07a_5d94_4e3f_ae91_5e8466250e78.slice/crio-72ddcd7d11530d7d4ed0e493029a59818b0e8bbcecb5ac8b13fadf812972238c WatchSource:0}: Error finding container 72ddcd7d11530d7d4ed0e493029a59818b0e8bbcecb5ac8b13fadf812972238c: Status 404 returned error can't find the container with id 72ddcd7d11530d7d4ed0e493029a59818b0e8bbcecb5ac8b13fadf812972238c Sep 11 14:24:06.915000 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.914971 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:24:06.915057 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.915035 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:24:06.915112 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:06.915062 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:24:07.569955 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:07.569923 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" event={"ID":"c8cad07a-5d94-4e3f-ae91-5e8466250e78","Type":"ContainerStarted","Data":"352c4c7fd009a2099491419f7a113c0cce1fd63cfdfe33a2a029c0986b11c969"} Sep 11 14:24:07.570373 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:07.569974 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" event={"ID":"c8cad07a-5d94-4e3f-ae91-5e8466250e78","Type":"ContainerStarted","Data":"72ddcd7d11530d7d4ed0e493029a59818b0e8bbcecb5ac8b13fadf812972238c"} Sep 11 14:24:07.613194 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:07.613141 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" podStartSLOduration=1.613124424 podStartE2EDuration="1.613124424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:24:07.610971704 +0000 UTC m=+1588.015952832" watchObservedRunningTime="2026-09-11 14:24:07.613124424 +0000 UTC m=+1588.018105552" Sep 11 14:24:07.618022 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:07.617990 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:07.623320 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:07.623297 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:08.575976 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:08.575945 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:08.577007 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:08.576987 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:21.298342 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:21.298306 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7708-e82940"] Sep 11 14:24:21.305106 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:21.303560 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-e82940" Sep 11 14:24:21.309107 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:21.306942 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4lz74\"" Sep 11 14:24:21.309107 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:21.308900 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7708-e82940"] Sep 11 14:24:21.425835 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:21.425793 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd2c7\" (UniqueName: \"kubernetes.io/projected/71f18e6a-ee86-407f-8aa3-0e9c930a99a1-kube-api-access-kd2c7\") pod \"test-curl-7708-e82940\" (UID: \"71f18e6a-ee86-407f-8aa3-0e9c930a99a1\") " pod="opendatahub/test-curl-7708-e82940" Sep 11 14:24:21.527236 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:21.527198 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd2c7\" (UniqueName: \"kubernetes.io/projected/71f18e6a-ee86-407f-8aa3-0e9c930a99a1-kube-api-access-kd2c7\") pod \"test-curl-7708-e82940\" (UID: \"71f18e6a-ee86-407f-8aa3-0e9c930a99a1\") " pod="opendatahub/test-curl-7708-e82940" Sep 11 14:24:21.538227 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:21.538190 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd2c7\" (UniqueName: \"kubernetes.io/projected/71f18e6a-ee86-407f-8aa3-0e9c930a99a1-kube-api-access-kd2c7\") pod \"test-curl-7708-e82940\" (UID: \"71f18e6a-ee86-407f-8aa3-0e9c930a99a1\") " pod="opendatahub/test-curl-7708-e82940" Sep 11 14:24:21.620557 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:21.620451 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-e82940" Sep 11 14:24:21.905760 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:21.905731 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7708-e82940"] Sep 11 14:24:21.923264 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:21.923229 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:24:22.662869 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:22.662837 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7708-e82940" event={"ID":"71f18e6a-ee86-407f-8aa3-0e9c930a99a1","Type":"ContainerStarted","Data":"83cda891605f8ba2fa480ed34f482493ad7b10133615e8ac6946df879cce1541"} Sep 11 14:24:25.688935 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:25.688896 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7708-e82940" event={"ID":"71f18e6a-ee86-407f-8aa3-0e9c930a99a1","Type":"ContainerStarted","Data":"f8dcc3e250f7fc4cc8b4cb34f8f3a838eff1d2515b8016db74d8a64ce4e981ff"} Sep 11 14:24:26.388024 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:26.387910 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7708-e82940" podStartSLOduration=2.125517389 podStartE2EDuration="5.387890062s" totalImagesPullingTime="3.262372673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:24:25.709772734 +0000 UTC m=+1606.114753862" watchObservedRunningTime="2026-09-11 14:24:26.387890062 +0000 UTC m=+1606.792871235" Sep 11 14:24:26.389572 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:26.389551 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7708-e82940"] Sep 11 14:24:26.395560 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:26.395531 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7708-e82940"] Sep 11 14:24:26.695203 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:24:26.695078 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7708-e82940" container="test-curl-7708-e82940" Sep 11 14:24:27.704526 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:27.704483 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7708-e82940" podUID="71f18e6a-ee86-407f-8aa3-0e9c930a99a1" containerName="test-curl-7708-e82940" containerID="cri-o://f8dcc3e250f7fc4cc8b4cb34f8f3a838eff1d2515b8016db74d8a64ce4e981ff" gracePeriod=2 Sep 11 14:24:27.707237 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:24:27.707208 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7708-e82940\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="71f18e6a-ee86-407f-8aa3-0e9c930a99a1" pod="opendatahub/test-curl-7708-e82940" Sep 11 14:24:29.720856 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:29.720831 2566 generic.go:356] "Generic (PLEG): container finished" podID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" containerID="352c4c7fd009a2099491419f7a113c0cce1fd63cfdfe33a2a029c0986b11c969" exitCode=0 Sep 11 14:24:29.721256 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:29.720916 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" event={"ID":"c8cad07a-5d94-4e3f-ae91-5e8466250e78","Type":"ContainerDied","Data":"352c4c7fd009a2099491419f7a113c0cce1fd63cfdfe33a2a029c0986b11c969"} Sep 11 14:24:29.721559 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:29.721539 2566 scope.go:117] "RemoveContainer" containerID="352c4c7fd009a2099491419f7a113c0cce1fd63cfdfe33a2a029c0986b11c969" Sep 11 14:24:29.722900 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:29.722873 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:24:29.722980 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:29.722943 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:24:29.722980 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:29.722975 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:24:29.901844 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:29.901816 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-e82940" Sep 11 14:24:29.915855 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:29.915818 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd2c7\" (UniqueName: \"kubernetes.io/projected/71f18e6a-ee86-407f-8aa3-0e9c930a99a1-kube-api-access-kd2c7\") pod \"71f18e6a-ee86-407f-8aa3-0e9c930a99a1\" (UID: \"71f18e6a-ee86-407f-8aa3-0e9c930a99a1\") " Sep 11 14:24:29.918335 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:29.918299 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f18e6a-ee86-407f-8aa3-0e9c930a99a1-kube-api-access-kd2c7" (OuterVolumeSpecName: "kube-api-access-kd2c7") pod "71f18e6a-ee86-407f-8aa3-0e9c930a99a1" (UID: "71f18e6a-ee86-407f-8aa3-0e9c930a99a1"). InnerVolumeSpecName "kube-api-access-kd2c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:24:30.017161 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:30.017122 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd2c7\" (UniqueName: \"kubernetes.io/projected/71f18e6a-ee86-407f-8aa3-0e9c930a99a1-kube-api-access-kd2c7\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:24:30.098816 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:30.098782 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71f18e6a-ee86-407f-8aa3-0e9c930a99a1" path="/var/lib/kubelet/pods/71f18e6a-ee86-407f-8aa3-0e9c930a99a1/volumes" Sep 11 14:24:30.732639 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:30.732606 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" event={"ID":"c8cad07a-5d94-4e3f-ae91-5e8466250e78","Type":"ContainerStarted","Data":"57b3c11831627a7848f12e171ea40e7ae55b721b4e295025ea19678ad5be82df"} Sep 11 14:24:30.733992 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:30.733966 2566 generic.go:356] "Generic (PLEG): container finished" podID="71f18e6a-ee86-407f-8aa3-0e9c930a99a1" containerID="f8dcc3e250f7fc4cc8b4cb34f8f3a838eff1d2515b8016db74d8a64ce4e981ff" exitCode=137 Sep 11 14:24:30.734133 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:30.734012 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-e82940" Sep 11 14:24:30.734133 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:30.734045 2566 scope.go:117] "RemoveContainer" containerID="f8dcc3e250f7fc4cc8b4cb34f8f3a838eff1d2515b8016db74d8a64ce4e981ff" Sep 11 14:24:30.743824 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:30.743806 2566 scope.go:117] "RemoveContainer" containerID="f8dcc3e250f7fc4cc8b4cb34f8f3a838eff1d2515b8016db74d8a64ce4e981ff" Sep 11 14:24:30.744123 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:24:30.744103 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8dcc3e250f7fc4cc8b4cb34f8f3a838eff1d2515b8016db74d8a64ce4e981ff\": container with ID starting with f8dcc3e250f7fc4cc8b4cb34f8f3a838eff1d2515b8016db74d8a64ce4e981ff not found: ID does not exist" containerID="f8dcc3e250f7fc4cc8b4cb34f8f3a838eff1d2515b8016db74d8a64ce4e981ff" Sep 11 14:24:30.744194 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:30.744136 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8dcc3e250f7fc4cc8b4cb34f8f3a838eff1d2515b8016db74d8a64ce4e981ff"} err="failed to get container status \"f8dcc3e250f7fc4cc8b4cb34f8f3a838eff1d2515b8016db74d8a64ce4e981ff\": rpc error: code = NotFound desc = could not find container \"f8dcc3e250f7fc4cc8b4cb34f8f3a838eff1d2515b8016db74d8a64ce4e981ff\": container with ID starting with f8dcc3e250f7fc4cc8b4cb34f8f3a838eff1d2515b8016db74d8a64ce4e981ff not found: ID does not exist" Sep 11 14:24:31.617619 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:31.617534 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:31.618309 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:31.618281 2566 patch_prober.go:28] interesting pod/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 11 14:24:31.618518 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:31.618493 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" podUID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 11 14:24:32.618785 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:32.618743 2566 patch_prober.go:28] interesting pod/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 11 14:24:32.620074 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:32.620038 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" podUID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 11 14:24:33.618232 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:33.618187 2566 patch_prober.go:28] interesting pod/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 11 14:24:33.618432 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:33.618256 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" podUID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 11 14:24:34.617733 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:34.617690 2566 patch_prober.go:28] interesting pod/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 11 14:24:34.618275 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:34.617762 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" podUID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 11 14:24:35.618379 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:35.618332 2566 patch_prober.go:28] interesting pod/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Sep 11 14:24:35.618887 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:35.618401 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" podUID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Sep 11 14:24:36.621850 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:36.621816 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:36.622265 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:36.622124 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:36.622729 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:36.622712 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:24:40.458192 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.458160 2566 scope.go:117] "RemoveContainer" containerID="6b5a9a1e7a634c90b1b00de50a41cbc5719d4f5fd8320a9790f418a52c92a90c" Sep 11 14:24:40.682552 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.682521 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5"] Sep 11 14:24:40.683000 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.682986 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71f18e6a-ee86-407f-8aa3-0e9c930a99a1" containerName="test-curl-7708-e82940" Sep 11 14:24:40.683053 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.683002 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="71f18e6a-ee86-407f-8aa3-0e9c930a99a1" containerName="test-curl-7708-e82940" Sep 11 14:24:40.683117 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.683072 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="71f18e6a-ee86-407f-8aa3-0e9c930a99a1" containerName="test-curl-7708-e82940" Sep 11 14:24:40.687615 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.687598 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.700658 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.700635 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bfa49a-openshift-default-dockercfg-fkpvj\"" Sep 11 14:24:40.714053 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.713990 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5"] Sep 11 14:24:40.832803 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.832774 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-podinfo\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.832803 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.832805 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-workload-certs\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.833043 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.832828 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-envoy\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.833043 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.832916 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-data\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.833043 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.832979 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37b3da8a-6b29-4299-b948-8080d3a00ae3-istiod-ca-cert\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.833043 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.832998 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-workload-socket\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.833247 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.833142 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-token\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.833247 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.833177 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-credential-socket\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.833247 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.833212 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qn97\" (UniqueName: \"kubernetes.io/projected/37b3da8a-6b29-4299-b948-8080d3a00ae3-kube-api-access-9qn97\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.934633 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.934595 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qn97\" (UniqueName: \"kubernetes.io/projected/37b3da8a-6b29-4299-b948-8080d3a00ae3-kube-api-access-9qn97\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.934633 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.934632 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-podinfo\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.934896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.934663 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-workload-certs\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.934896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.934715 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-envoy\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.934896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.934769 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-data\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.934896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.934817 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37b3da8a-6b29-4299-b948-8080d3a00ae3-istiod-ca-cert\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.934896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.934848 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-workload-socket\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.935200 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.934950 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-token\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.935200 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.934988 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-credential-socket\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.935200 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.935061 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-workload-certs\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.935351 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.935241 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-data\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.935406 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.935373 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-credential-socket\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.935406 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.935395 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-workload-socket\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.935795 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.935774 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37b3da8a-6b29-4299-b948-8080d3a00ae3-istiod-ca-cert\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.937454 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.937432 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-envoy\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.937585 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.937542 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-podinfo\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.949703 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.949683 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qn97\" (UniqueName: \"kubernetes.io/projected/37b3da8a-6b29-4299-b948-8080d3a00ae3-kube-api-access-9qn97\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.950240 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.950225 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-token\") pod \"e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:40.997565 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:40.997536 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:41.288141 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:41.286922 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5"] Sep 11 14:24:41.295448 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:41.295414 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:24:41.295560 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:41.295481 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:24:41.295560 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:41.295506 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:24:41.812584 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:41.812549 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" event={"ID":"37b3da8a-6b29-4299-b948-8080d3a00ae3","Type":"ContainerStarted","Data":"f9968d8fd5d252fb02b43cd8e8d8adf68c990265fb268612c184ddc453256eb5"} Sep 11 14:24:41.812997 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:41.812609 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" event={"ID":"37b3da8a-6b29-4299-b948-8080d3a00ae3","Type":"ContainerStarted","Data":"7879d0dd9afd966f2f94cdc185e0bbed9f8b97c341d0dd491de69db4de2efe94"} Sep 11 14:24:41.852791 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:41.852740 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" podStartSLOduration=1.8527230220000002 podStartE2EDuration="1.852723022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:24:41.847688562 +0000 UTC m=+1622.252669712" watchObservedRunningTime="2026-09-11 14:24:41.852723022 +0000 UTC m=+1622.257704149" Sep 11 14:24:41.998037 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:41.997999 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:43.004614 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:43.004583 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:43.825068 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:43.824981 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:24:43.825925 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:24:43.825904 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:25:05.979419 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:05.979386 2566 generic.go:356] "Generic (PLEG): container finished" podID="37b3da8a-6b29-4299-b948-8080d3a00ae3" containerID="f9968d8fd5d252fb02b43cd8e8d8adf68c990265fb268612c184ddc453256eb5" exitCode=0 Sep 11 14:25:05.979824 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:05.979465 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" event={"ID":"37b3da8a-6b29-4299-b948-8080d3a00ae3","Type":"ContainerDied","Data":"f9968d8fd5d252fb02b43cd8e8d8adf68c990265fb268612c184ddc453256eb5"} Sep 11 14:25:05.979919 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:05.979905 2566 scope.go:117] "RemoveContainer" containerID="f9968d8fd5d252fb02b43cd8e8d8adf68c990265fb268612c184ddc453256eb5" Sep 11 14:25:05.981171 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:05.981142 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:25:05.981307 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:05.981225 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:25:05.981307 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:05.981271 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:25:06.988915 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:06.988871 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" event={"ID":"37b3da8a-6b29-4299-b948-8080d3a00ae3","Type":"ContainerStarted","Data":"13bd18b46d7f30d686cffe2106fd45c84d1f45166833cab0f039ce2e01097d7d"} Sep 11 14:25:06.997807 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:06.997777 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:25:07.998884 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:07.998850 2566 patch_prober.go:28] interesting pod/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Sep 11 14:25:07.999511 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:07.999478 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" podUID="37b3da8a-6b29-4299-b948-8080d3a00ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Sep 11 14:25:08.998371 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:08.998327 2566 patch_prober.go:28] interesting pod/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Sep 11 14:25:08.998599 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:08.998407 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" podUID="37b3da8a-6b29-4299-b948-8080d3a00ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Sep 11 14:25:09.998181 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:09.998141 2566 patch_prober.go:28] interesting pod/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Sep 11 14:25:09.998673 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:09.998212 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" podUID="37b3da8a-6b29-4299-b948-8080d3a00ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Sep 11 14:25:11.009605 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:11.009553 2566 patch_prober.go:28] interesting pod/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Sep 11 14:25:11.010131 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:11.009630 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" podUID="37b3da8a-6b29-4299-b948-8080d3a00ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Sep 11 14:25:11.998873 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:11.998829 2566 patch_prober.go:28] interesting pod/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Sep 11 14:25:11.999103 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:11.998899 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" podUID="37b3da8a-6b29-4299-b948-8080d3a00ae3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Sep 11 14:25:13.002660 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:13.002629 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:25:13.003156 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:13.003130 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:25:13.003971 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:13.003952 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:25:17.268372 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:17.268334 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9"] Sep 11 14:25:17.272253 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:17.272236 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9" Sep 11 14:25:17.276018 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:17.276000 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:25:17.283966 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:17.283941 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9"] Sep 11 14:25:17.422728 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:17.422693 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8917aef-3fec-4abc-9bba-e66ca4574e3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9\" (UID: \"b8917aef-3fec-4abc-9bba-e66ca4574e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9" Sep 11 14:25:17.523687 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:17.523601 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8917aef-3fec-4abc-9bba-e66ca4574e3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9\" (UID: \"b8917aef-3fec-4abc-9bba-e66ca4574e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9" Sep 11 14:25:17.524258 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:17.524239 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8917aef-3fec-4abc-9bba-e66ca4574e3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9\" (UID: \"b8917aef-3fec-4abc-9bba-e66ca4574e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9" Sep 11 14:25:17.583523 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:17.583488 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9" Sep 11 14:25:17.862292 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:17.862248 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9"] Sep 11 14:25:17.863471 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:25:17.863446 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8917aef_3fec_4abc_9bba_e66ca4574e3f.slice/crio-5c5f65e1ec79558c318647ed6272ad6dfb4c9f5b59fd83726beda5e394eac9d3 WatchSource:0}: Error finding container 5c5f65e1ec79558c318647ed6272ad6dfb4c9f5b59fd83726beda5e394eac9d3: Status 404 returned error can't find the container with id 5c5f65e1ec79558c318647ed6272ad6dfb4c9f5b59fd83726beda5e394eac9d3 Sep 11 14:25:18.079677 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:18.079639 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9" event={"ID":"b8917aef-3fec-4abc-9bba-e66ca4574e3f","Type":"ContainerStarted","Data":"5c5f65e1ec79558c318647ed6272ad6dfb4c9f5b59fd83726beda5e394eac9d3"} Sep 11 14:25:19.088430 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:19.088397 2566 generic.go:356] "Generic (PLEG): container finished" podID="b8917aef-3fec-4abc-9bba-e66ca4574e3f" containerID="68fa4273328d7ffc00df92189b8c5024aa12f150432a3492348379ad2006a693" exitCode=0 Sep 11 14:25:19.088835 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:19.088456 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9" event={"ID":"b8917aef-3fec-4abc-9bba-e66ca4574e3f","Type":"ContainerDied","Data":"68fa4273328d7ffc00df92189b8c5024aa12f150432a3492348379ad2006a693"} Sep 11 14:25:20.238939 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:20.238903 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9" Sep 11 14:25:20.348315 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:20.348231 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8917aef-3fec-4abc-9bba-e66ca4574e3f-maas-api-service-ca\") pod \"b8917aef-3fec-4abc-9bba-e66ca4574e3f\" (UID: \"b8917aef-3fec-4abc-9bba-e66ca4574e3f\") " Sep 11 14:25:20.348647 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:20.348626 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8917aef-3fec-4abc-9bba-e66ca4574e3f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b8917aef-3fec-4abc-9bba-e66ca4574e3f" (UID: "b8917aef-3fec-4abc-9bba-e66ca4574e3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:25:20.449658 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:20.449624 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8917aef-3fec-4abc-9bba-e66ca4574e3f-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:25:21.107439 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:21.107409 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9" Sep 11 14:25:21.107644 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:21.107404 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9" event={"ID":"b8917aef-3fec-4abc-9bba-e66ca4574e3f","Type":"ContainerDied","Data":"5c5f65e1ec79558c318647ed6272ad6dfb4c9f5b59fd83726beda5e394eac9d3"} Sep 11 14:25:21.107644 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:21.107543 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c5f65e1ec79558c318647ed6272ad6dfb4c9f5b59fd83726beda5e394eac9d3" Sep 11 14:25:39.394135 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:39.393979 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9"] Sep 11 14:25:39.405295 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:39.405259 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d52edb-h9mp9"] Sep 11 14:25:40.098445 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:40.098363 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8917aef-3fec-4abc-9bba-e66ca4574e3f" path="/var/lib/kubelet/pods/b8917aef-3fec-4abc-9bba-e66ca4574e3f/volumes" Sep 11 14:25:40.487981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:40.487950 2566 scope.go:117] "RemoveContainer" containerID="acde355de24d22baa142abbf7caffc58a19ebfdd466f36c5f19fb5b94d3355c2" Sep 11 14:25:40.497305 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:40.497282 2566 scope.go:117] "RemoveContainer" containerID="bf4683252b052fc09e430fc7920a871d015e0477f287440a13740432cb84d1d1" Sep 11 14:25:40.507629 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:40.507605 2566 scope.go:117] "RemoveContainer" containerID="994aa45202fd3011ba354947a0026fef5cd6457ea754442c81da82ffa292c0f1" Sep 11 14:25:47.639595 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:47.639560 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg"] Sep 11 14:25:47.640013 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:47.639792 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" podUID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" containerName="istio-proxy" containerID="cri-o://57b3c11831627a7848f12e171ea40e7ae55b721b4e295025ea19678ad5be82df" gracePeriod=30 Sep 11 14:25:47.978727 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:47.978691 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm"] Sep 11 14:25:47.979156 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:47.979143 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8917aef-3fec-4abc-9bba-e66ca4574e3f" containerName="revoke-keys" Sep 11 14:25:47.979209 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:47.979158 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8917aef-3fec-4abc-9bba-e66ca4574e3f" containerName="revoke-keys" Sep 11 14:25:47.979246 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:47.979231 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8917aef-3fec-4abc-9bba-e66ca4574e3f" containerName="revoke-keys" Sep 11 14:25:47.982471 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:47.982445 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm" Sep 11 14:25:47.985066 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:47.985045 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:25:47.994520 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:47.994494 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm"] Sep 11 14:25:48.027056 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:48.027024 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ed4ca-2e80-4c7c-9237-1300df30c811-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm\" (UID: \"0a3ed4ca-2e80-4c7c-9237-1300df30c811\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm" Sep 11 14:25:48.127858 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:48.127831 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ed4ca-2e80-4c7c-9237-1300df30c811-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm\" (UID: \"0a3ed4ca-2e80-4c7c-9237-1300df30c811\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm" Sep 11 14:25:48.128490 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:48.128467 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ed4ca-2e80-4c7c-9237-1300df30c811-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm\" (UID: \"0a3ed4ca-2e80-4c7c-9237-1300df30c811\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm" Sep 11 14:25:48.294018 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:48.293932 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm" Sep 11 14:25:48.582207 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:48.582145 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm"] Sep 11 14:25:48.582940 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:25:48.582913 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3ed4ca_2e80_4c7c_9237_1300df30c811.slice/crio-de202e424f9dbeb11fe61fe76f8d5f5a8df2ec8a277b0d39475207196c40591b WatchSource:0}: Error finding container de202e424f9dbeb11fe61fe76f8d5f5a8df2ec8a277b0d39475207196c40591b: Status 404 returned error can't find the container with id de202e424f9dbeb11fe61fe76f8d5f5a8df2ec8a277b0d39475207196c40591b Sep 11 14:25:49.290775 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:49.290740 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm" event={"ID":"0a3ed4ca-2e80-4c7c-9237-1300df30c811","Type":"ContainerStarted","Data":"cdb97a1a62bec6be60deae8af5022a9156cbb274596bb36482d485cc20a38bcc"} Sep 11 14:25:49.291213 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:49.290799 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm" event={"ID":"0a3ed4ca-2e80-4c7c-9237-1300df30c811","Type":"ContainerStarted","Data":"de202e424f9dbeb11fe61fe76f8d5f5a8df2ec8a277b0d39475207196c40591b"} Sep 11 14:25:49.308910 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:49.308836 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm" podStartSLOduration=2.30881657 podStartE2EDuration="2.30881657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:25:49.30723664 +0000 UTC m=+1689.712217781" watchObservedRunningTime="2026-09-11 14:25:49.30881657 +0000 UTC m=+1689.713797700" Sep 11 14:25:50.298700 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:50.298660 2566 generic.go:356] "Generic (PLEG): container finished" podID="0a3ed4ca-2e80-4c7c-9237-1300df30c811" containerID="cdb97a1a62bec6be60deae8af5022a9156cbb274596bb36482d485cc20a38bcc" exitCode=0 Sep 11 14:25:50.299132 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:50.298754 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm" event={"ID":"0a3ed4ca-2e80-4c7c-9237-1300df30c811","Type":"ContainerDied","Data":"cdb97a1a62bec6be60deae8af5022a9156cbb274596bb36482d485cc20a38bcc"} Sep 11 14:25:51.449978 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:51.449952 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm" Sep 11 14:25:51.566772 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:51.566676 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ed4ca-2e80-4c7c-9237-1300df30c811-maas-api-service-ca\") pod \"0a3ed4ca-2e80-4c7c-9237-1300df30c811\" (UID: \"0a3ed4ca-2e80-4c7c-9237-1300df30c811\") " Sep 11 14:25:51.567055 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:51.567029 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3ed4ca-2e80-4c7c-9237-1300df30c811-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a3ed4ca-2e80-4c7c-9237-1300df30c811" (UID: "0a3ed4ca-2e80-4c7c-9237-1300df30c811"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:25:51.624993 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:51.624952 2566 patch_prober.go:28] interesting pod/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:25:51.625178 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:51.625023 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" podUID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:25:51.667818 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:51.667782 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3ed4ca-2e80-4c7c-9237-1300df30c811-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:25:52.316217 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.316187 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm" Sep 11 14:25:52.316388 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.316236 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm" event={"ID":"0a3ed4ca-2e80-4c7c-9237-1300df30c811","Type":"ContainerDied","Data":"de202e424f9dbeb11fe61fe76f8d5f5a8df2ec8a277b0d39475207196c40591b"} Sep 11 14:25:52.316388 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.316289 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de202e424f9dbeb11fe61fe76f8d5f5a8df2ec8a277b0d39475207196c40591b" Sep 11 14:25:52.900708 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.900647 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:25:52.980364 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.980326 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-workload-certs\") pod \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " Sep 11 14:25:52.980364 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.980368 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-envoy\") pod \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " Sep 11 14:25:52.980671 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.980392 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-data\") pod \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " Sep 11 14:25:52.980671 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.980434 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-workload-socket\") pod \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " Sep 11 14:25:52.980671 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.980454 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-credential-socket\") pod \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " Sep 11 14:25:52.980671 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.980475 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-podinfo\") pod \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " Sep 11 14:25:52.980671 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.980503 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zzpf\" (UniqueName: \"kubernetes.io/projected/c8cad07a-5d94-4e3f-ae91-5e8466250e78-kube-api-access-4zzpf\") pod \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " Sep 11 14:25:52.980671 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.980524 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-token\") pod \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " Sep 11 14:25:52.980671 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.980609 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istiod-ca-cert\") pod \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\" (UID: \"c8cad07a-5d94-4e3f-ae91-5e8466250e78\") " Sep 11 14:25:52.981002 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.980679 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c8cad07a-5d94-4e3f-ae91-5e8466250e78" (UID: "c8cad07a-5d94-4e3f-ae91-5e8466250e78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:25:52.981002 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.980774 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c8cad07a-5d94-4e3f-ae91-5e8466250e78" (UID: "c8cad07a-5d94-4e3f-ae91-5e8466250e78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:25:52.981002 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.980811 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-data" (OuterVolumeSpecName: "istio-data") pod "c8cad07a-5d94-4e3f-ae91-5e8466250e78" (UID: "c8cad07a-5d94-4e3f-ae91-5e8466250e78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:25:52.981002 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.980848 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c8cad07a-5d94-4e3f-ae91-5e8466250e78" (UID: "c8cad07a-5d94-4e3f-ae91-5e8466250e78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:25:52.981281 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.981023 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c8cad07a-5d94-4e3f-ae91-5e8466250e78" (UID: "c8cad07a-5d94-4e3f-ae91-5e8466250e78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:25:52.981281 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.981124 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:25:52.981281 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.981145 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:25:52.981281 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.981161 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:25:52.981281 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.981174 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:25:52.981281 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.981186 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:25:52.982815 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.982789 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c8cad07a-5d94-4e3f-ae91-5e8466250e78" (UID: "c8cad07a-5d94-4e3f-ae91-5e8466250e78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:25:52.982959 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.982938 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c8cad07a-5d94-4e3f-ae91-5e8466250e78" (UID: "c8cad07a-5d94-4e3f-ae91-5e8466250e78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:25:52.983014 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.982999 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-token" (OuterVolumeSpecName: "istio-token") pod "c8cad07a-5d94-4e3f-ae91-5e8466250e78" (UID: "c8cad07a-5d94-4e3f-ae91-5e8466250e78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:25:52.983049 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:52.983017 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8cad07a-5d94-4e3f-ae91-5e8466250e78-kube-api-access-4zzpf" (OuterVolumeSpecName: "kube-api-access-4zzpf") pod "c8cad07a-5d94-4e3f-ae91-5e8466250e78" (UID: "c8cad07a-5d94-4e3f-ae91-5e8466250e78"). InnerVolumeSpecName "kube-api-access-4zzpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:25:53.082618 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.082579 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:25:53.082618 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.082610 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:25:53.082618 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.082619 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zzpf\" (UniqueName: \"kubernetes.io/projected/c8cad07a-5d94-4e3f-ae91-5e8466250e78-kube-api-access-4zzpf\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:25:53.082856 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.082629 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8cad07a-5d94-4e3f-ae91-5e8466250e78-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:25:53.324294 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.324259 2566 generic.go:356] "Generic (PLEG): container finished" podID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" containerID="57b3c11831627a7848f12e171ea40e7ae55b721b4e295025ea19678ad5be82df" exitCode=0 Sep 11 14:25:53.324447 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.324313 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" Sep 11 14:25:53.324447 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.324339 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" event={"ID":"c8cad07a-5d94-4e3f-ae91-5e8466250e78","Type":"ContainerDied","Data":"57b3c11831627a7848f12e171ea40e7ae55b721b4e295025ea19678ad5be82df"} Sep 11 14:25:53.324447 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.324402 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg" event={"ID":"c8cad07a-5d94-4e3f-ae91-5e8466250e78","Type":"ContainerDied","Data":"72ddcd7d11530d7d4ed0e493029a59818b0e8bbcecb5ac8b13fadf812972238c"} Sep 11 14:25:53.324447 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.324425 2566 scope.go:117] "RemoveContainer" containerID="57b3c11831627a7848f12e171ea40e7ae55b721b4e295025ea19678ad5be82df" Sep 11 14:25:53.334882 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.334863 2566 scope.go:117] "RemoveContainer" containerID="352c4c7fd009a2099491419f7a113c0cce1fd63cfdfe33a2a029c0986b11c969" Sep 11 14:25:53.344268 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.344247 2566 scope.go:117] "RemoveContainer" containerID="57b3c11831627a7848f12e171ea40e7ae55b721b4e295025ea19678ad5be82df" Sep 11 14:25:53.344563 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:25:53.344545 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b3c11831627a7848f12e171ea40e7ae55b721b4e295025ea19678ad5be82df\": container with ID starting with 57b3c11831627a7848f12e171ea40e7ae55b721b4e295025ea19678ad5be82df not found: ID does not exist" containerID="57b3c11831627a7848f12e171ea40e7ae55b721b4e295025ea19678ad5be82df" Sep 11 14:25:53.344615 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.344573 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b3c11831627a7848f12e171ea40e7ae55b721b4e295025ea19678ad5be82df"} err="failed to get container status \"57b3c11831627a7848f12e171ea40e7ae55b721b4e295025ea19678ad5be82df\": rpc error: code = NotFound desc = could not find container \"57b3c11831627a7848f12e171ea40e7ae55b721b4e295025ea19678ad5be82df\": container with ID starting with 57b3c11831627a7848f12e171ea40e7ae55b721b4e295025ea19678ad5be82df not found: ID does not exist" Sep 11 14:25:53.344615 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.344592 2566 scope.go:117] "RemoveContainer" containerID="352c4c7fd009a2099491419f7a113c0cce1fd63cfdfe33a2a029c0986b11c969" Sep 11 14:25:53.344871 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:25:53.344845 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352c4c7fd009a2099491419f7a113c0cce1fd63cfdfe33a2a029c0986b11c969\": container with ID starting with 352c4c7fd009a2099491419f7a113c0cce1fd63cfdfe33a2a029c0986b11c969 not found: ID does not exist" containerID="352c4c7fd009a2099491419f7a113c0cce1fd63cfdfe33a2a029c0986b11c969" Sep 11 14:25:53.344915 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.344878 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352c4c7fd009a2099491419f7a113c0cce1fd63cfdfe33a2a029c0986b11c969"} err="failed to get container status \"352c4c7fd009a2099491419f7a113c0cce1fd63cfdfe33a2a029c0986b11c969\": rpc error: code = NotFound desc = could not find container \"352c4c7fd009a2099491419f7a113c0cce1fd63cfdfe33a2a029c0986b11c969\": container with ID starting with 352c4c7fd009a2099491419f7a113c0cce1fd63cfdfe33a2a029c0986b11c969 not found: ID does not exist" Sep 11 14:25:53.372597 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.372562 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg"] Sep 11 14:25:53.380201 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:53.380171 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d52edb-openshift-default-59bb6c8c96-jv5bg"] Sep 11 14:25:54.098647 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:25:54.098600 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" path="/var/lib/kubelet/pods/c8cad07a-5d94-4e3f-ae91-5e8466250e78/volumes" Sep 11 14:26:04.082500 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:04.082418 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm"] Sep 11 14:26:04.087043 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:04.087020 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bfa49a-w82sm"] Sep 11 14:26:04.099352 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:04.099315 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a3ed4ca-2e80-4c7c-9237-1300df30c811" path="/var/lib/kubelet/pods/0a3ed4ca-2e80-4c7c-9237-1300df30c811/volumes" Sep 11 14:26:11.256394 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.256363 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5"] Sep 11 14:26:11.256959 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.256574 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" podUID="37b3da8a-6b29-4299-b948-8080d3a00ae3" containerName="istio-proxy" containerID="cri-o://13bd18b46d7f30d686cffe2106fd45c84d1f45166833cab0f039ce2e01097d7d" gracePeriod=30 Sep 11 14:26:11.624696 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.624615 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg"] Sep 11 14:26:11.625061 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.625049 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" containerName="istio-proxy" Sep 11 14:26:11.625164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.625063 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" containerName="istio-proxy" Sep 11 14:26:11.625164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.625079 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a3ed4ca-2e80-4c7c-9237-1300df30c811" containerName="revoke-keys" Sep 11 14:26:11.625164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.625099 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a3ed4ca-2e80-4c7c-9237-1300df30c811" containerName="revoke-keys" Sep 11 14:26:11.625164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.625116 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" containerName="istio-proxy" Sep 11 14:26:11.625164 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.625123 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" containerName="istio-proxy" Sep 11 14:26:11.625338 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.625191 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a3ed4ca-2e80-4c7c-9237-1300df30c811" containerName="revoke-keys" Sep 11 14:26:11.625338 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.625202 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" containerName="istio-proxy" Sep 11 14:26:11.625338 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.625208 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8cad07a-5d94-4e3f-ae91-5e8466250e78" containerName="istio-proxy" Sep 11 14:26:11.628296 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.628281 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg" Sep 11 14:26:11.631520 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.631497 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:26:11.639758 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.639732 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg"] Sep 11 14:26:11.674961 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.674922 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e34f1cb2-1526-4d4a-a362-5e030b737ace-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg\" (UID: \"e34f1cb2-1526-4d4a-a362-5e030b737ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg" Sep 11 14:26:11.776174 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.776141 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e34f1cb2-1526-4d4a-a362-5e030b737ace-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg\" (UID: \"e34f1cb2-1526-4d4a-a362-5e030b737ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg" Sep 11 14:26:11.776765 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.776743 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e34f1cb2-1526-4d4a-a362-5e030b737ace-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg\" (UID: \"e34f1cb2-1526-4d4a-a362-5e030b737ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg" Sep 11 14:26:11.940826 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:11.940732 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg" Sep 11 14:26:12.257289 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:12.257257 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg"] Sep 11 14:26:12.258801 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:26:12.258770 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34f1cb2_1526_4d4a_a362_5e030b737ace.slice/crio-631e8f6d93c90c67b36b2ffb895209ae18c53ff5e98bb70b64ff42838e8a319c WatchSource:0}: Error finding container 631e8f6d93c90c67b36b2ffb895209ae18c53ff5e98bb70b64ff42838e8a319c: Status 404 returned error can't find the container with id 631e8f6d93c90c67b36b2ffb895209ae18c53ff5e98bb70b64ff42838e8a319c Sep 11 14:26:12.450331 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:12.450294 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg" event={"ID":"e34f1cb2-1526-4d4a-a362-5e030b737ace","Type":"ContainerStarted","Data":"631e8f6d93c90c67b36b2ffb895209ae18c53ff5e98bb70b64ff42838e8a319c"} Sep 11 14:26:13.005373 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:13.005336 2566 patch_prober.go:28] interesting pod/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:26:13.005551 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:13.005406 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" podUID="37b3da8a-6b29-4299-b948-8080d3a00ae3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:26:13.458121 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:13.458012 2566 generic.go:356] "Generic (PLEG): container finished" podID="e34f1cb2-1526-4d4a-a362-5e030b737ace" containerID="f08354a23a824da92e61452ae03ebdf70cfa6e457588467af4dfda5412f39211" exitCode=0 Sep 11 14:26:13.458503 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:13.458118 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg" event={"ID":"e34f1cb2-1526-4d4a-a362-5e030b737ace","Type":"ContainerDied","Data":"f08354a23a824da92e61452ae03ebdf70cfa6e457588467af4dfda5412f39211"} Sep 11 14:26:14.613068 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:14.613041 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg" Sep 11 14:26:14.706059 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:14.706021 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e34f1cb2-1526-4d4a-a362-5e030b737ace-maas-api-service-ca\") pod \"e34f1cb2-1526-4d4a-a362-5e030b737ace\" (UID: \"e34f1cb2-1526-4d4a-a362-5e030b737ace\") " Sep 11 14:26:14.706391 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:14.706369 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34f1cb2-1526-4d4a-a362-5e030b737ace-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e34f1cb2-1526-4d4a-a362-5e030b737ace" (UID: "e34f1cb2-1526-4d4a-a362-5e030b737ace"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:26:14.807534 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:14.807502 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e34f1cb2-1526-4d4a-a362-5e030b737ace-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:15.472891 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:15.472856 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg" Sep 11 14:26:15.472891 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:15.472881 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg" event={"ID":"e34f1cb2-1526-4d4a-a362-5e030b737ace","Type":"ContainerDied","Data":"631e8f6d93c90c67b36b2ffb895209ae18c53ff5e98bb70b64ff42838e8a319c"} Sep 11 14:26:15.473134 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:15.472940 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="631e8f6d93c90c67b36b2ffb895209ae18c53ff5e98bb70b64ff42838e8a319c" Sep 11 14:26:16.481496 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.481464 2566 generic.go:356] "Generic (PLEG): container finished" podID="37b3da8a-6b29-4299-b948-8080d3a00ae3" containerID="13bd18b46d7f30d686cffe2106fd45c84d1f45166833cab0f039ce2e01097d7d" exitCode=0 Sep 11 14:26:16.481985 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.481534 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" event={"ID":"37b3da8a-6b29-4299-b948-8080d3a00ae3","Type":"ContainerDied","Data":"13bd18b46d7f30d686cffe2106fd45c84d1f45166833cab0f039ce2e01097d7d"} Sep 11 14:26:16.481985 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.481599 2566 scope.go:117] "RemoveContainer" containerID="f9968d8fd5d252fb02b43cd8e8d8adf68c990265fb268612c184ddc453256eb5" Sep 11 14:26:16.510956 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:26:16.510929 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9968d8fd5d252fb02b43cd8e8d8adf68c990265fb268612c184ddc453256eb5\": container with ID starting with f9968d8fd5d252fb02b43cd8e8d8adf68c990265fb268612c184ddc453256eb5 not found: ID does not exist" containerID="f9968d8fd5d252fb02b43cd8e8d8adf68c990265fb268612c184ddc453256eb5" Sep 11 14:26:16.514377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.514357 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:26:16.626955 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.626865 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-podinfo\") pod \"37b3da8a-6b29-4299-b948-8080d3a00ae3\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " Sep 11 14:26:16.626955 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.626906 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-envoy\") pod \"37b3da8a-6b29-4299-b948-8080d3a00ae3\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " Sep 11 14:26:16.626955 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.626923 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37b3da8a-6b29-4299-b948-8080d3a00ae3-istiod-ca-cert\") pod \"37b3da8a-6b29-4299-b948-8080d3a00ae3\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " Sep 11 14:26:16.627305 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.626970 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-credential-socket\") pod \"37b3da8a-6b29-4299-b948-8080d3a00ae3\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " Sep 11 14:26:16.627305 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.627052 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qn97\" (UniqueName: \"kubernetes.io/projected/37b3da8a-6b29-4299-b948-8080d3a00ae3-kube-api-access-9qn97\") pod \"37b3da8a-6b29-4299-b948-8080d3a00ae3\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " Sep 11 14:26:16.627305 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.627118 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-workload-socket\") pod \"37b3da8a-6b29-4299-b948-8080d3a00ae3\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " Sep 11 14:26:16.627305 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.627183 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-workload-certs\") pod \"37b3da8a-6b29-4299-b948-8080d3a00ae3\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " Sep 11 14:26:16.627305 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.627207 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-data\") pod \"37b3da8a-6b29-4299-b948-8080d3a00ae3\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " Sep 11 14:26:16.627613 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.627338 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "37b3da8a-6b29-4299-b948-8080d3a00ae3" (UID: "37b3da8a-6b29-4299-b948-8080d3a00ae3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:26:16.627613 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.627453 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "37b3da8a-6b29-4299-b948-8080d3a00ae3" (UID: "37b3da8a-6b29-4299-b948-8080d3a00ae3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:26:16.627613 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.627492 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "37b3da8a-6b29-4299-b948-8080d3a00ae3" (UID: "37b3da8a-6b29-4299-b948-8080d3a00ae3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:26:16.627613 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.627493 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b3da8a-6b29-4299-b948-8080d3a00ae3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "37b3da8a-6b29-4299-b948-8080d3a00ae3" (UID: "37b3da8a-6b29-4299-b948-8080d3a00ae3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:26:16.627613 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.627518 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-token\") pod \"37b3da8a-6b29-4299-b948-8080d3a00ae3\" (UID: \"37b3da8a-6b29-4299-b948-8080d3a00ae3\") " Sep 11 14:26:16.628044 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.627706 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-data" (OuterVolumeSpecName: "istio-data") pod "37b3da8a-6b29-4299-b948-8080d3a00ae3" (UID: "37b3da8a-6b29-4299-b948-8080d3a00ae3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:26:16.628220 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.628168 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:16.628220 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.628194 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:16.628220 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.628210 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:16.628410 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.628224 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:16.628410 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.628237 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37b3da8a-6b29-4299-b948-8080d3a00ae3-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:16.629455 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.629425 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "37b3da8a-6b29-4299-b948-8080d3a00ae3" (UID: "37b3da8a-6b29-4299-b948-8080d3a00ae3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:26:16.629455 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.629431 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b3da8a-6b29-4299-b948-8080d3a00ae3-kube-api-access-9qn97" (OuterVolumeSpecName: "kube-api-access-9qn97") pod "37b3da8a-6b29-4299-b948-8080d3a00ae3" (UID: "37b3da8a-6b29-4299-b948-8080d3a00ae3"). InnerVolumeSpecName "kube-api-access-9qn97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:26:16.629626 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.629470 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "37b3da8a-6b29-4299-b948-8080d3a00ae3" (UID: "37b3da8a-6b29-4299-b948-8080d3a00ae3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:26:16.629690 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.629622 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-token" (OuterVolumeSpecName: "istio-token") pod "37b3da8a-6b29-4299-b948-8080d3a00ae3" (UID: "37b3da8a-6b29-4299-b948-8080d3a00ae3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:26:16.729613 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.729572 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qn97\" (UniqueName: \"kubernetes.io/projected/37b3da8a-6b29-4299-b948-8080d3a00ae3-kube-api-access-9qn97\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:16.729613 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.729606 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:16.729613 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.729619 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:16.729861 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:16.729631 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37b3da8a-6b29-4299-b948-8080d3a00ae3-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:17.489762 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:17.489732 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" Sep 11 14:26:17.490229 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:17.489779 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5" event={"ID":"37b3da8a-6b29-4299-b948-8080d3a00ae3","Type":"ContainerDied","Data":"7879d0dd9afd966f2f94cdc185e0bbed9f8b97c341d0dd491de69db4de2efe94"} Sep 11 14:26:17.490229 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:17.489838 2566 scope.go:117] "RemoveContainer" containerID="13bd18b46d7f30d686cffe2106fd45c84d1f45166833cab0f039ce2e01097d7d" Sep 11 14:26:17.513712 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:17.513686 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5"] Sep 11 14:26:17.521204 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:17.521165 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bfa49a-openshift-default-76bfb4b488-rq6s5"] Sep 11 14:26:18.097805 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:18.097755 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b3da8a-6b29-4299-b948-8080d3a00ae3" path="/var/lib/kubelet/pods/37b3da8a-6b29-4299-b948-8080d3a00ae3/volumes" Sep 11 14:26:22.705769 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:22.705732 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg"] Sep 11 14:26:22.737393 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:22.737361 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fb993c-jthfg"] Sep 11 14:26:23.473395 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.473363 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7708-a85f8e"] Sep 11 14:26:23.473782 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.473769 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e34f1cb2-1526-4d4a-a362-5e030b737ace" containerName="revoke-keys" Sep 11 14:26:23.473829 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.473786 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e34f1cb2-1526-4d4a-a362-5e030b737ace" containerName="revoke-keys" Sep 11 14:26:23.473829 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.473821 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b3da8a-6b29-4299-b948-8080d3a00ae3" containerName="istio-proxy" Sep 11 14:26:23.473829 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.473828 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b3da8a-6b29-4299-b948-8080d3a00ae3" containerName="istio-proxy" Sep 11 14:26:23.473930 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.473837 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b3da8a-6b29-4299-b948-8080d3a00ae3" containerName="istio-proxy" Sep 11 14:26:23.473930 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.473842 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b3da8a-6b29-4299-b948-8080d3a00ae3" containerName="istio-proxy" Sep 11 14:26:23.473930 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.473903 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e34f1cb2-1526-4d4a-a362-5e030b737ace" containerName="revoke-keys" Sep 11 14:26:23.473930 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.473913 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b3da8a-6b29-4299-b948-8080d3a00ae3" containerName="istio-proxy" Sep 11 14:26:23.473930 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.473918 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b3da8a-6b29-4299-b948-8080d3a00ae3" containerName="istio-proxy" Sep 11 14:26:23.478600 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.478578 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-a85f8e" Sep 11 14:26:23.481568 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.481548 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4lz74\"" Sep 11 14:26:23.487442 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.487419 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7708-a85f8e"] Sep 11 14:26:23.594718 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.594685 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zc46\" (UniqueName: \"kubernetes.io/projected/53992568-56eb-40cf-9c27-d898b10f8fff-kube-api-access-2zc46\") pod \"test-curl-7708-a85f8e\" (UID: \"53992568-56eb-40cf-9c27-d898b10f8fff\") " pod="opendatahub/test-curl-7708-a85f8e" Sep 11 14:26:23.702025 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.697119 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zc46\" (UniqueName: \"kubernetes.io/projected/53992568-56eb-40cf-9c27-d898b10f8fff-kube-api-access-2zc46\") pod \"test-curl-7708-a85f8e\" (UID: \"53992568-56eb-40cf-9c27-d898b10f8fff\") " pod="opendatahub/test-curl-7708-a85f8e" Sep 11 14:26:23.712059 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.712023 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zc46\" (UniqueName: \"kubernetes.io/projected/53992568-56eb-40cf-9c27-d898b10f8fff-kube-api-access-2zc46\") pod \"test-curl-7708-a85f8e\" (UID: \"53992568-56eb-40cf-9c27-d898b10f8fff\") " pod="opendatahub/test-curl-7708-a85f8e" Sep 11 14:26:23.788719 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:23.788664 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-a85f8e" Sep 11 14:26:24.082538 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:24.082509 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7708-a85f8e"] Sep 11 14:26:24.084440 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:26:24.084416 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53992568_56eb_40cf_9c27_d898b10f8fff.slice/crio-5de03b407105109f44248c18093d6529e8e94f1157b4f9e0776e51224dd3bb20 WatchSource:0}: Error finding container 5de03b407105109f44248c18093d6529e8e94f1157b4f9e0776e51224dd3bb20: Status 404 returned error can't find the container with id 5de03b407105109f44248c18093d6529e8e94f1157b4f9e0776e51224dd3bb20 Sep 11 14:26:24.098159 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:24.098131 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e34f1cb2-1526-4d4a-a362-5e030b737ace" path="/var/lib/kubelet/pods/e34f1cb2-1526-4d4a-a362-5e030b737ace/volumes" Sep 11 14:26:24.537601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:24.537564 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7708-a85f8e" event={"ID":"53992568-56eb-40cf-9c27-d898b10f8fff","Type":"ContainerStarted","Data":"fc2838b369a581e327052c52b3e36873446fb62e5c94e7fe5b3962d6f2777970"} Sep 11 14:26:24.537804 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:24.537608 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7708-a85f8e" event={"ID":"53992568-56eb-40cf-9c27-d898b10f8fff","Type":"ContainerStarted","Data":"5de03b407105109f44248c18093d6529e8e94f1157b4f9e0776e51224dd3bb20"} Sep 11 14:26:24.578641 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:24.578588 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7708-a85f8e" podStartSLOduration=1.5785738660000002 podStartE2EDuration="1.578573866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:26:24.577127164 +0000 UTC m=+1724.982108292" watchObservedRunningTime="2026-09-11 14:26:24.578573866 +0000 UTC m=+1724.983554994" Sep 11 14:26:25.227337 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:25.227303 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7708-a85f8e"] Sep 11 14:26:25.231137 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:25.231107 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7708-a85f8e"] Sep 11 14:26:25.544008 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:26:25.543976 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7708-a85f8e" container="test-curl-7708-a85f8e" Sep 11 14:26:26.549963 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:26.549924 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7708-a85f8e" podUID="53992568-56eb-40cf-9c27-d898b10f8fff" containerName="test-curl-7708-a85f8e" containerID="cri-o://fc2838b369a581e327052c52b3e36873446fb62e5c94e7fe5b3962d6f2777970" gracePeriod=2 Sep 11 14:26:28.734634 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:28.734605 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-a85f8e" Sep 11 14:26:28.849321 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:28.849223 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zc46\" (UniqueName: \"kubernetes.io/projected/53992568-56eb-40cf-9c27-d898b10f8fff-kube-api-access-2zc46\") pod \"53992568-56eb-40cf-9c27-d898b10f8fff\" (UID: \"53992568-56eb-40cf-9c27-d898b10f8fff\") " Sep 11 14:26:28.851806 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:28.851777 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53992568-56eb-40cf-9c27-d898b10f8fff-kube-api-access-2zc46" (OuterVolumeSpecName: "kube-api-access-2zc46") pod "53992568-56eb-40cf-9c27-d898b10f8fff" (UID: "53992568-56eb-40cf-9c27-d898b10f8fff"). InnerVolumeSpecName "kube-api-access-2zc46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:26:28.950769 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:28.950733 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zc46\" (UniqueName: \"kubernetes.io/projected/53992568-56eb-40cf-9c27-d898b10f8fff-kube-api-access-2zc46\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:29.569242 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:29.569210 2566 generic.go:356] "Generic (PLEG): container finished" podID="53992568-56eb-40cf-9c27-d898b10f8fff" containerID="fc2838b369a581e327052c52b3e36873446fb62e5c94e7fe5b3962d6f2777970" exitCode=137 Sep 11 14:26:29.569407 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:29.569314 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7708-a85f8e" Sep 11 14:26:29.569407 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:29.569330 2566 scope.go:117] "RemoveContainer" containerID="fc2838b369a581e327052c52b3e36873446fb62e5c94e7fe5b3962d6f2777970" Sep 11 14:26:29.579196 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:29.579169 2566 scope.go:117] "RemoveContainer" containerID="fc2838b369a581e327052c52b3e36873446fb62e5c94e7fe5b3962d6f2777970" Sep 11 14:26:29.579463 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:26:29.579443 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc2838b369a581e327052c52b3e36873446fb62e5c94e7fe5b3962d6f2777970\": container with ID starting with fc2838b369a581e327052c52b3e36873446fb62e5c94e7fe5b3962d6f2777970 not found: ID does not exist" containerID="fc2838b369a581e327052c52b3e36873446fb62e5c94e7fe5b3962d6f2777970" Sep 11 14:26:29.579536 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:29.579474 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2838b369a581e327052c52b3e36873446fb62e5c94e7fe5b3962d6f2777970"} err="failed to get container status \"fc2838b369a581e327052c52b3e36873446fb62e5c94e7fe5b3962d6f2777970\": rpc error: code = NotFound desc = could not find container \"fc2838b369a581e327052c52b3e36873446fb62e5c94e7fe5b3962d6f2777970\": container with ID starting with fc2838b369a581e327052c52b3e36873446fb62e5c94e7fe5b3962d6f2777970 not found: ID does not exist" Sep 11 14:26:29.579714 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:26:29.579697 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7708-a85f8e\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="53992568-56eb-40cf-9c27-d898b10f8fff" pod="opendatahub/test-curl-7708-a85f8e" Sep 11 14:26:30.099436 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:26:30.099390 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7708-a85f8e\" is forbidden: User \"system:node:ip-10-0-136-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-92.ec2.internal' and this object" podUID="53992568-56eb-40cf-9c27-d898b10f8fff" pod="opendatahub/test-curl-7708-a85f8e" Sep 11 14:26:30.102857 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:30.102817 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53992568-56eb-40cf-9c27-d898b10f8fff" path="/var/lib/kubelet/pods/53992568-56eb-40cf-9c27-d898b10f8fff/volumes" Sep 11 14:26:30.650175 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:30.650143 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb"] Sep 11 14:26:30.650423 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:30.650376 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" podUID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerName="istio-proxy" containerID="cri-o://0b9c9cb7efa8438ad6a24ade743d3ca39789ebbb5e4b45a396df5455cc86ba64" gracePeriod=30 Sep 11 14:26:30.986348 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:30.986320 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t"] Sep 11 14:26:30.986768 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:30.986754 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53992568-56eb-40cf-9c27-d898b10f8fff" containerName="test-curl-7708-a85f8e" Sep 11 14:26:30.986812 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:30.986771 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="53992568-56eb-40cf-9c27-d898b10f8fff" containerName="test-curl-7708-a85f8e" Sep 11 14:26:30.986854 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:30.986845 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="53992568-56eb-40cf-9c27-d898b10f8fff" containerName="test-curl-7708-a85f8e" Sep 11 14:26:30.991379 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:30.991365 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t" Sep 11 14:26:30.994676 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:30.994652 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:26:30.999341 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:30.999316 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t"] Sep 11 14:26:31.071806 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:31.071770 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t\" (UID: \"9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t" Sep 11 14:26:31.172819 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:31.172781 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t\" (UID: \"9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t" Sep 11 14:26:31.173403 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:31.173385 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t\" (UID: \"9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t" Sep 11 14:26:31.303748 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:31.303671 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t" Sep 11 14:26:31.590328 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:31.590297 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t"] Sep 11 14:26:31.591603 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:26:31.591579 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3258cc_34ac_4cf5_a8a2_a8149b5ae72f.slice/crio-1a7968235eb6b70fa455dc7709868d57e592e575a67873ef4b9f4a8f100814fb WatchSource:0}: Error finding container 1a7968235eb6b70fa455dc7709868d57e592e575a67873ef4b9f4a8f100814fb: Status 404 returned error can't find the container with id 1a7968235eb6b70fa455dc7709868d57e592e575a67873ef4b9f4a8f100814fb Sep 11 14:26:32.591196 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:32.591113 2566 generic.go:356] "Generic (PLEG): container finished" podID="9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f" containerID="ba77bfdc6249c6b979e7cf329c2085703be87af8c9dc64522b7b96f7b0d4dfb5" exitCode=0 Sep 11 14:26:32.591196 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:32.591151 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t" event={"ID":"9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f","Type":"ContainerDied","Data":"ba77bfdc6249c6b979e7cf329c2085703be87af8c9dc64522b7b96f7b0d4dfb5"} Sep 11 14:26:32.591196 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:32.591191 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t" event={"ID":"9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f","Type":"ContainerStarted","Data":"1a7968235eb6b70fa455dc7709868d57e592e575a67873ef4b9f4a8f100814fb"} Sep 11 14:26:33.738694 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:33.738670 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t" Sep 11 14:26:33.900294 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:33.900201 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f-maas-api-service-ca\") pod \"9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f\" (UID: \"9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f\") " Sep 11 14:26:33.900625 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:33.900597 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f" (UID: "9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:26:34.001820 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:34.001785 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:34.605427 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:34.605396 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t" Sep 11 14:26:34.605609 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:34.605434 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t" event={"ID":"9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f","Type":"ContainerDied","Data":"1a7968235eb6b70fa455dc7709868d57e592e575a67873ef4b9f4a8f100814fb"} Sep 11 14:26:34.605609 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:34.605486 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a7968235eb6b70fa455dc7709868d57e592e575a67873ef4b9f4a8f100814fb" Sep 11 14:26:35.917451 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:35.917396 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:26:36.020000 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.019971 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-podinfo\") pod \"a5f713d3-d284-4778-b474-1df1a1d06d17\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " Sep 11 14:26:36.020186 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.020013 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-envoy\") pod \"a5f713d3-d284-4778-b474-1df1a1d06d17\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " Sep 11 14:26:36.020186 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.020052 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-token\") pod \"a5f713d3-d284-4778-b474-1df1a1d06d17\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " Sep 11 14:26:36.020186 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.020110 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8smg8\" (UniqueName: \"kubernetes.io/projected/a5f713d3-d284-4778-b474-1df1a1d06d17-kube-api-access-8smg8\") pod \"a5f713d3-d284-4778-b474-1df1a1d06d17\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " Sep 11 14:26:36.020186 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.020144 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-workload-certs\") pod \"a5f713d3-d284-4778-b474-1df1a1d06d17\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " Sep 11 14:26:36.020465 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.020284 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-workload-socket\") pod \"a5f713d3-d284-4778-b474-1df1a1d06d17\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " Sep 11 14:26:36.020465 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.020401 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-credential-socket\") pod \"a5f713d3-d284-4778-b474-1df1a1d06d17\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " Sep 11 14:26:36.020465 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.020435 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f713d3-d284-4778-b474-1df1a1d06d17-istiod-ca-cert\") pod \"a5f713d3-d284-4778-b474-1df1a1d06d17\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " Sep 11 14:26:36.020465 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.020442 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a5f713d3-d284-4778-b474-1df1a1d06d17" (UID: "a5f713d3-d284-4778-b474-1df1a1d06d17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:26:36.020685 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.020512 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-data\") pod \"a5f713d3-d284-4778-b474-1df1a1d06d17\" (UID: \"a5f713d3-d284-4778-b474-1df1a1d06d17\") " Sep 11 14:26:36.020685 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.020658 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a5f713d3-d284-4778-b474-1df1a1d06d17" (UID: "a5f713d3-d284-4778-b474-1df1a1d06d17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:26:36.021009 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.020783 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f713d3-d284-4778-b474-1df1a1d06d17-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a5f713d3-d284-4778-b474-1df1a1d06d17" (UID: "a5f713d3-d284-4778-b474-1df1a1d06d17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:26:36.021072 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.021028 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-data" (OuterVolumeSpecName: "istio-data") pod "a5f713d3-d284-4778-b474-1df1a1d06d17" (UID: "a5f713d3-d284-4778-b474-1df1a1d06d17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:26:36.021167 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.021078 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:36.021167 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.021107 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a5f713d3-d284-4778-b474-1df1a1d06d17" (UID: "a5f713d3-d284-4778-b474-1df1a1d06d17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:26:36.021167 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.021122 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:36.022523 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.022501 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f713d3-d284-4778-b474-1df1a1d06d17-kube-api-access-8smg8" (OuterVolumeSpecName: "kube-api-access-8smg8") pod "a5f713d3-d284-4778-b474-1df1a1d06d17" (UID: "a5f713d3-d284-4778-b474-1df1a1d06d17"). InnerVolumeSpecName "kube-api-access-8smg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:26:36.022523 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.022503 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a5f713d3-d284-4778-b474-1df1a1d06d17" (UID: "a5f713d3-d284-4778-b474-1df1a1d06d17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:26:36.022662 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.022539 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a5f713d3-d284-4778-b474-1df1a1d06d17" (UID: "a5f713d3-d284-4778-b474-1df1a1d06d17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:26:36.022662 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.022593 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-token" (OuterVolumeSpecName: "istio-token") pod "a5f713d3-d284-4778-b474-1df1a1d06d17" (UID: "a5f713d3-d284-4778-b474-1df1a1d06d17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:26:36.122188 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.122158 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:36.122188 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.122183 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8smg8\" (UniqueName: \"kubernetes.io/projected/a5f713d3-d284-4778-b474-1df1a1d06d17-kube-api-access-8smg8\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:36.122188 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.122193 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:36.122389 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.122201 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f713d3-d284-4778-b474-1df1a1d06d17-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:36.122389 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.122209 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:36.122389 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.122216 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:36.122389 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.122223 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f713d3-d284-4778-b474-1df1a1d06d17-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:36.619051 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.619015 2566 generic.go:356] "Generic (PLEG): container finished" podID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerID="0b9c9cb7efa8438ad6a24ade743d3ca39789ebbb5e4b45a396df5455cc86ba64" exitCode=0 Sep 11 14:26:36.619244 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.619065 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" Sep 11 14:26:36.619244 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.619124 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" event={"ID":"a5f713d3-d284-4778-b474-1df1a1d06d17","Type":"ContainerDied","Data":"0b9c9cb7efa8438ad6a24ade743d3ca39789ebbb5e4b45a396df5455cc86ba64"} Sep 11 14:26:36.619244 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.619180 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb" event={"ID":"a5f713d3-d284-4778-b474-1df1a1d06d17","Type":"ContainerDied","Data":"260f90a63ea20ace76ce028c1868bbc251fd9b4f321a6b30c3cc2346d665c6ed"} Sep 11 14:26:36.619244 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.619204 2566 scope.go:117] "RemoveContainer" containerID="0b9c9cb7efa8438ad6a24ade743d3ca39789ebbb5e4b45a396df5455cc86ba64" Sep 11 14:26:36.628751 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.628731 2566 scope.go:117] "RemoveContainer" containerID="a26be4307612c0e28d6434346cae6b5f33ecb22a415a72d3fb134c8a0588c1fe" Sep 11 14:26:36.637498 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.637476 2566 scope.go:117] "RemoveContainer" containerID="0b9c9cb7efa8438ad6a24ade743d3ca39789ebbb5e4b45a396df5455cc86ba64" Sep 11 14:26:36.637777 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:26:36.637758 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b9c9cb7efa8438ad6a24ade743d3ca39789ebbb5e4b45a396df5455cc86ba64\": container with ID starting with 0b9c9cb7efa8438ad6a24ade743d3ca39789ebbb5e4b45a396df5455cc86ba64 not found: ID does not exist" containerID="0b9c9cb7efa8438ad6a24ade743d3ca39789ebbb5e4b45a396df5455cc86ba64" Sep 11 14:26:36.637820 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.637788 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9c9cb7efa8438ad6a24ade743d3ca39789ebbb5e4b45a396df5455cc86ba64"} err="failed to get container status \"0b9c9cb7efa8438ad6a24ade743d3ca39789ebbb5e4b45a396df5455cc86ba64\": rpc error: code = NotFound desc = could not find container \"0b9c9cb7efa8438ad6a24ade743d3ca39789ebbb5e4b45a396df5455cc86ba64\": container with ID starting with 0b9c9cb7efa8438ad6a24ade743d3ca39789ebbb5e4b45a396df5455cc86ba64 not found: ID does not exist" Sep 11 14:26:36.637820 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.637808 2566 scope.go:117] "RemoveContainer" containerID="a26be4307612c0e28d6434346cae6b5f33ecb22a415a72d3fb134c8a0588c1fe" Sep 11 14:26:36.638035 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:26:36.638019 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a26be4307612c0e28d6434346cae6b5f33ecb22a415a72d3fb134c8a0588c1fe\": container with ID starting with a26be4307612c0e28d6434346cae6b5f33ecb22a415a72d3fb134c8a0588c1fe not found: ID does not exist" containerID="a26be4307612c0e28d6434346cae6b5f33ecb22a415a72d3fb134c8a0588c1fe" Sep 11 14:26:36.638077 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:36.638043 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26be4307612c0e28d6434346cae6b5f33ecb22a415a72d3fb134c8a0588c1fe"} err="failed to get container status \"a26be4307612c0e28d6434346cae6b5f33ecb22a415a72d3fb134c8a0588c1fe\": rpc error: code = NotFound desc = could not find container \"a26be4307612c0e28d6434346cae6b5f33ecb22a415a72d3fb134c8a0588c1fe\": container with ID starting with a26be4307612c0e28d6434346cae6b5f33ecb22a415a72d3fb134c8a0588c1fe not found: ID does not exist" Sep 11 14:26:38.596939 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:38.596905 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb"] Sep 11 14:26:39.981628 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:39.981592 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fb993c-openshift-default-54c78c6946-67xfb"] Sep 11 14:26:40.099512 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:40.099484 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5f713d3-d284-4778-b474-1df1a1d06d17" path="/var/lib/kubelet/pods/a5f713d3-d284-4778-b474-1df1a1d06d17/volumes" Sep 11 14:26:44.901780 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:44.901742 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t"] Sep 11 14:26:44.906914 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:44.906889 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-335a68-4mj7t"] Sep 11 14:26:46.098416 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:46.098377 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f" path="/var/lib/kubelet/pods/9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f/volumes" Sep 11 14:26:53.012284 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:53.012253 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz"] Sep 11 14:26:53.012775 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:53.012483 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" podUID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" containerName="istio-proxy" containerID="cri-o://a4cc6298c85ce6bad4c3d54707e5c8237da0c27374f9dde45f34b6675772e940" gracePeriod=30 Sep 11 14:26:55.869606 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:55.869571 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:26:55.870053 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:55.869632 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" podUID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:26:58.281467 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.281406 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:26:58.324234 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324195 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-data\") pod \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " Sep 11 14:26:58.324389 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324268 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-credential-socket\") pod \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " Sep 11 14:26:58.324389 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324294 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-workload-socket\") pod \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " Sep 11 14:26:58.324389 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324321 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-workload-certs\") pod \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " Sep 11 14:26:58.324564 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324404 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-envoy\") pod \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " Sep 11 14:26:58.324564 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324441 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istiod-ca-cert\") pod \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " Sep 11 14:26:58.324564 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324483 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7djh\" (UniqueName: \"kubernetes.io/projected/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-kube-api-access-r7djh\") pod \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " Sep 11 14:26:58.324564 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324518 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-podinfo\") pod \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " Sep 11 14:26:58.324564 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324528 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" (UID: "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:26:58.324808 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324587 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-token\") pod \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\" (UID: \"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd\") " Sep 11 14:26:58.324808 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324617 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" (UID: "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:26:58.324808 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324614 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" (UID: "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:26:58.324808 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324693 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" (UID: "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:26:58.325004 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324896 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" (UID: "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:26:58.325004 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324909 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:58.325004 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324929 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:58.325004 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324944 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:58.325004 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.324958 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:58.326995 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.326944 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-kube-api-access-r7djh" (OuterVolumeSpecName: "kube-api-access-r7djh") pod "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" (UID: "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd"). InnerVolumeSpecName "kube-api-access-r7djh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:26:58.326995 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.326955 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" (UID: "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:26:58.327226 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.326991 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" (UID: "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:26:58.327344 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.327326 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" (UID: "79ad9660-2319-4d8f-a3f7-d3a61e50d6dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:26:58.425986 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.425948 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:58.425986 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.425978 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:58.425986 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.425991 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7djh\" (UniqueName: \"kubernetes.io/projected/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-kube-api-access-r7djh\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:58.426279 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.426005 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:58.426279 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.426016 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:26:58.773724 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.773684 2566 generic.go:356] "Generic (PLEG): container finished" podID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" containerID="a4cc6298c85ce6bad4c3d54707e5c8237da0c27374f9dde45f34b6675772e940" exitCode=0 Sep 11 14:26:58.773895 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.773739 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" Sep 11 14:26:58.773895 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.773771 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" event={"ID":"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd","Type":"ContainerDied","Data":"a4cc6298c85ce6bad4c3d54707e5c8237da0c27374f9dde45f34b6675772e940"} Sep 11 14:26:58.773895 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.773835 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz" event={"ID":"79ad9660-2319-4d8f-a3f7-d3a61e50d6dd","Type":"ContainerDied","Data":"3eb8ba65ce8828fd53a65ecab38828839072f063cf4af94a4d07f75d482680b1"} Sep 11 14:26:58.773895 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.773859 2566 scope.go:117] "RemoveContainer" containerID="a4cc6298c85ce6bad4c3d54707e5c8237da0c27374f9dde45f34b6675772e940" Sep 11 14:26:58.784438 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.784420 2566 scope.go:117] "RemoveContainer" containerID="024f45de152d457f7913936ac60e306f3af70bf9dd5ef5e7b7a9b38763d76d9c" Sep 11 14:26:58.793658 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.793641 2566 scope.go:117] "RemoveContainer" containerID="a4cc6298c85ce6bad4c3d54707e5c8237da0c27374f9dde45f34b6675772e940" Sep 11 14:26:58.793954 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:26:58.793936 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4cc6298c85ce6bad4c3d54707e5c8237da0c27374f9dde45f34b6675772e940\": container with ID starting with a4cc6298c85ce6bad4c3d54707e5c8237da0c27374f9dde45f34b6675772e940 not found: ID does not exist" containerID="a4cc6298c85ce6bad4c3d54707e5c8237da0c27374f9dde45f34b6675772e940" Sep 11 14:26:58.794011 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.793969 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4cc6298c85ce6bad4c3d54707e5c8237da0c27374f9dde45f34b6675772e940"} err="failed to get container status \"a4cc6298c85ce6bad4c3d54707e5c8237da0c27374f9dde45f34b6675772e940\": rpc error: code = NotFound desc = could not find container \"a4cc6298c85ce6bad4c3d54707e5c8237da0c27374f9dde45f34b6675772e940\": container with ID starting with a4cc6298c85ce6bad4c3d54707e5c8237da0c27374f9dde45f34b6675772e940 not found: ID does not exist" Sep 11 14:26:58.794011 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.793987 2566 scope.go:117] "RemoveContainer" containerID="024f45de152d457f7913936ac60e306f3af70bf9dd5ef5e7b7a9b38763d76d9c" Sep 11 14:26:58.794279 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:26:58.794262 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024f45de152d457f7913936ac60e306f3af70bf9dd5ef5e7b7a9b38763d76d9c\": container with ID starting with 024f45de152d457f7913936ac60e306f3af70bf9dd5ef5e7b7a9b38763d76d9c not found: ID does not exist" containerID="024f45de152d457f7913936ac60e306f3af70bf9dd5ef5e7b7a9b38763d76d9c" Sep 11 14:26:58.794347 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.794288 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024f45de152d457f7913936ac60e306f3af70bf9dd5ef5e7b7a9b38763d76d9c"} err="failed to get container status \"024f45de152d457f7913936ac60e306f3af70bf9dd5ef5e7b7a9b38763d76d9c\": rpc error: code = NotFound desc = could not find container \"024f45de152d457f7913936ac60e306f3af70bf9dd5ef5e7b7a9b38763d76d9c\": container with ID starting with 024f45de152d457f7913936ac60e306f3af70bf9dd5ef5e7b7a9b38763d76d9c not found: ID does not exist" Sep 11 14:26:58.822370 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.822336 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz"] Sep 11 14:26:58.827051 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:26:58.827025 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-335a68-openshift-default-dcf4788d-mjgbz"] Sep 11 14:27:00.098073 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:27:00.098038 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" path="/var/lib/kubelet/pods/79ad9660-2319-4d8f-a3f7-d3a61e50d6dd/volumes" Sep 11 14:27:40.240671 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:27:40.240623 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:27:40.243803 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:27:40.243781 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:28:13.960729 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.960696 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7"] Sep 11 14:28:13.961354 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.961328 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerName="istio-proxy" Sep 11 14:28:13.961354 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.961355 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerName="istio-proxy" Sep 11 14:28:13.961499 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.961372 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f" containerName="revoke-keys" Sep 11 14:28:13.961499 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.961378 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f" containerName="revoke-keys" Sep 11 14:28:13.961499 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.961395 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerName="istio-proxy" Sep 11 14:28:13.961499 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.961401 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerName="istio-proxy" Sep 11 14:28:13.961499 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.961407 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" containerName="istio-proxy" Sep 11 14:28:13.961499 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.961414 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" containerName="istio-proxy" Sep 11 14:28:13.961499 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.961429 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" containerName="istio-proxy" Sep 11 14:28:13.961499 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.961435 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" containerName="istio-proxy" Sep 11 14:28:13.961795 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.961511 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerName="istio-proxy" Sep 11 14:28:13.961795 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.961520 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" containerName="istio-proxy" Sep 11 14:28:13.961795 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.961529 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5f713d3-d284-4778-b474-1df1a1d06d17" containerName="istio-proxy" Sep 11 14:28:13.961795 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.961537 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f3258cc-34ac-4cf5-a8a2-a8149b5ae72f" containerName="revoke-keys" Sep 11 14:28:13.964278 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.964261 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:13.967460 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.967441 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-30989f-openshift-default-dockercfg-swvbz\"" Sep 11 14:28:13.976207 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:13.976179 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7"] Sep 11 14:28:14.001971 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.001937 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-workload-socket\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.002165 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.001988 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-podinfo\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.002165 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.002052 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istiod-ca-cert\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.002165 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.002156 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4w4c\" (UniqueName: \"kubernetes.io/projected/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-kube-api-access-b4w4c\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.002334 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.002204 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-envoy\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.002334 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.002255 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-token\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.002445 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.002359 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-data\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.002445 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.002412 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-workload-certs\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.002548 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.002471 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-credential-socket\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.103687 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.103652 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-workload-socket\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.103889 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.103701 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-podinfo\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.103889 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.103726 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istiod-ca-cert\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.103889 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.103764 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4w4c\" (UniqueName: \"kubernetes.io/projected/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-kube-api-access-b4w4c\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.103889 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.103805 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-envoy\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.103889 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.103850 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-token\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.104211 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.103898 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-data\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.104211 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.103935 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-workload-certs\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.104211 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.103977 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-credential-socket\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.104381 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.104347 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-credential-socket\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.104435 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.104422 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-workload-socket\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.104615 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.104590 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-data\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.104708 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.104608 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istiod-ca-cert\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.104769 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.104725 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-workload-certs\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.106634 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.106601 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-podinfo\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.106768 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.106753 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-envoy\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.113124 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.113080 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-token\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.113817 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.113795 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4w4c\" (UniqueName: \"kubernetes.io/projected/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-kube-api-access-b4w4c\") pod \"e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.275745 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.275716 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:14.557529 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.557502 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7"] Sep 11 14:28:14.559075 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:28:14.559047 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa5f5af_9d1f_4095_96e5_0e64f52fde1f.slice/crio-8b583bfc631cd1bd2e550188c7fb9d793c42f8347fe94cb1aae547432e0e8868 WatchSource:0}: Error finding container 8b583bfc631cd1bd2e550188c7fb9d793c42f8347fe94cb1aae547432e0e8868: Status 404 returned error can't find the container with id 8b583bfc631cd1bd2e550188c7fb9d793c42f8347fe94cb1aae547432e0e8868 Sep 11 14:28:14.561376 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.561344 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:28:14.561450 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.561413 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:28:14.561450 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:14.561441 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:28:15.249135 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:15.249070 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" event={"ID":"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f","Type":"ContainerStarted","Data":"a2a7acb0e73dce2481fc470da080ca9674305f60b90d87879c4a6d2a133a6027"} Sep 11 14:28:15.249542 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:15.249160 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" event={"ID":"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f","Type":"ContainerStarted","Data":"8b583bfc631cd1bd2e550188c7fb9d793c42f8347fe94cb1aae547432e0e8868"} Sep 11 14:28:15.276512 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:15.276483 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:15.281624 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:15.281601 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:15.290875 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:15.290833 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" podStartSLOduration=2.290819461 podStartE2EDuration="2.290819461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:15.287216872 +0000 UTC m=+1835.692198011" watchObservedRunningTime="2026-09-11 14:28:15.290819461 +0000 UTC m=+1835.695800589" Sep 11 14:28:16.255161 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:16.255132 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:16.256198 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:16.256177 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:32.296315 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:32.296269 2566 patch_prober.go:28] interesting pod/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 14:28:32.296723 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:32.296341 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" podUID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 14:28:34.375856 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:34.375754 2566 generic.go:356] "Generic (PLEG): container finished" podID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" containerID="a2a7acb0e73dce2481fc470da080ca9674305f60b90d87879c4a6d2a133a6027" exitCode=0 Sep 11 14:28:34.375856 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:34.375832 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" event={"ID":"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f","Type":"ContainerDied","Data":"a2a7acb0e73dce2481fc470da080ca9674305f60b90d87879c4a6d2a133a6027"} Sep 11 14:28:34.376340 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:34.376251 2566 scope.go:117] "RemoveContainer" containerID="a2a7acb0e73dce2481fc470da080ca9674305f60b90d87879c4a6d2a133a6027" Sep 11 14:28:34.377371 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:34.377341 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:28:34.377431 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:34.377415 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:28:34.377465 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:34.377454 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:28:35.384550 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:35.384515 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" event={"ID":"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f","Type":"ContainerStarted","Data":"91cf779aff8c26e26008995b9ba9fa7674ee5dd74529eaf1b26d4d8f5bb944b8"} Sep 11 14:28:36.276546 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:36.276506 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:36.276785 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:36.276761 2566 patch_prober.go:28] interesting pod/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Sep 11 14:28:36.276857 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:36.276837 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" podUID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Sep 11 14:28:37.277070 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:37.277028 2566 patch_prober.go:28] interesting pod/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Sep 11 14:28:37.277460 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:37.277115 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" podUID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Sep 11 14:28:38.276549 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:38.276509 2566 patch_prober.go:28] interesting pod/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Sep 11 14:28:38.276741 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:38.276575 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" podUID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Sep 11 14:28:39.277233 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:39.277198 2566 patch_prober.go:28] interesting pod/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Sep 11 14:28:39.277631 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:39.277268 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" podUID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Sep 11 14:28:40.277138 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:40.277103 2566 patch_prober.go:28] interesting pod/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Sep 11 14:28:40.277351 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:40.277164 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" podUID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Sep 11 14:28:40.582616 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:40.582530 2566 scope.go:117] "RemoveContainer" containerID="7ef1ce14d6ddb92a72649cc2aa5950d37f36e78b0d3ed8737fe5f4ac839283d9" Sep 11 14:28:41.280279 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:41.280245 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:41.280747 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:41.280539 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:41.281222 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:41.281205 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:28:43.286298 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.286266 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h"] Sep 11 14:28:43.286771 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.286758 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ad9660-2319-4d8f-a3f7-d3a61e50d6dd" containerName="istio-proxy" Sep 11 14:28:43.289696 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.289677 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.292371 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.292346 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b7d522-openshift-default-dockercfg-2smvp\"" Sep 11 14:28:43.294175 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.294150 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-workload-socket\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.294304 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.294189 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-podinfo\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.294304 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.294215 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-credential-socket\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.294304 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.294266 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-data\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.294304 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.294299 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-token\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.294479 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.294345 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-envoy\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.294479 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.294402 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-workload-certs\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.294479 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.294424 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17db22a-8bd6-4571-a575-1b160cf26ff2-istiod-ca-cert\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.294479 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.294469 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7glc\" (UniqueName: \"kubernetes.io/projected/a17db22a-8bd6-4571-a575-1b160cf26ff2-kube-api-access-b7glc\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.307046 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.307016 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h"] Sep 11 14:28:43.395232 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.395193 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-envoy\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.396142 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.396114 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-workload-certs\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.396341 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.396325 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17db22a-8bd6-4571-a575-1b160cf26ff2-istiod-ca-cert\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.396484 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.396471 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7glc\" (UniqueName: \"kubernetes.io/projected/a17db22a-8bd6-4571-a575-1b160cf26ff2-kube-api-access-b7glc\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.396598 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.396586 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-workload-socket\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.396697 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.396684 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-podinfo\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.396840 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.396802 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-credential-socket\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.397035 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.397016 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-data\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.397174 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.397140 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17db22a-8bd6-4571-a575-1b160cf26ff2-istiod-ca-cert\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.397332 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.397116 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-workload-socket\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.397464 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.396516 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-workload-certs\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.397560 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.397183 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-token\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.397684 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.397343 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-data\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.397770 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.397056 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-credential-socket\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.400457 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.400426 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-envoy\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.405666 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.404693 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-podinfo\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.410480 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.410454 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-token\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.411268 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.411244 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7glc\" (UniqueName: \"kubernetes.io/projected/a17db22a-8bd6-4571-a575-1b160cf26ff2-kube-api-access-b7glc\") pod \"e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.602469 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.602362 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:43.886881 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.886797 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h"] Sep 11 14:28:43.889778 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:28:43.889744 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17db22a_8bd6_4571_a575_1b160cf26ff2.slice/crio-2b92be7ddf30d934f7e782703f585307a4ec8c3ae1943d1f46c0cc4ae6f517c6 WatchSource:0}: Error finding container 2b92be7ddf30d934f7e782703f585307a4ec8c3ae1943d1f46c0cc4ae6f517c6: Status 404 returned error can't find the container with id 2b92be7ddf30d934f7e782703f585307a4ec8c3ae1943d1f46c0cc4ae6f517c6 Sep 11 14:28:43.892217 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.892189 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:28:43.892293 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.892251 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:28:43.892293 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:43.892280 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:28:44.463449 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:44.463407 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" event={"ID":"a17db22a-8bd6-4571-a575-1b160cf26ff2","Type":"ContainerStarted","Data":"0db5ed0db24752dbb7a28a15290d0eec6a7691139876c50bea7acf129bf24631"} Sep 11 14:28:44.463912 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:44.463480 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" event={"ID":"a17db22a-8bd6-4571-a575-1b160cf26ff2","Type":"ContainerStarted","Data":"2b92be7ddf30d934f7e782703f585307a4ec8c3ae1943d1f46c0cc4ae6f517c6"} Sep 11 14:28:44.489770 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:44.489719 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" podStartSLOduration=1.489701568 podStartE2EDuration="1.489701568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:44.48816975 +0000 UTC m=+1864.893150877" watchObservedRunningTime="2026-09-11 14:28:44.489701568 +0000 UTC m=+1864.894682696" Sep 11 14:28:44.602625 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:44.602588 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:44.609251 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:44.609217 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:45.470114 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:45.470057 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:28:45.471079 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:28:45.471058 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:29:05.605739 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:05.605700 2566 generic.go:356] "Generic (PLEG): container finished" podID="a17db22a-8bd6-4571-a575-1b160cf26ff2" containerID="0db5ed0db24752dbb7a28a15290d0eec6a7691139876c50bea7acf129bf24631" exitCode=0 Sep 11 14:29:05.606191 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:05.605767 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" event={"ID":"a17db22a-8bd6-4571-a575-1b160cf26ff2","Type":"ContainerDied","Data":"0db5ed0db24752dbb7a28a15290d0eec6a7691139876c50bea7acf129bf24631"} Sep 11 14:29:05.606242 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:05.606230 2566 scope.go:117] "RemoveContainer" containerID="0db5ed0db24752dbb7a28a15290d0eec6a7691139876c50bea7acf129bf24631" Sep 11 14:29:05.607349 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:05.607318 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:29:05.607457 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:05.607387 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:29:05.607457 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:05.607419 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:29:06.624429 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:06.624392 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" event={"ID":"a17db22a-8bd6-4571-a575-1b160cf26ff2","Type":"ContainerStarted","Data":"28250acfcf630c16e7e9d26e06567e97e7f49a9288388da4a424d08e3c0ee8e7"} Sep 11 14:29:07.602575 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:07.602537 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:29:07.603253 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:07.603226 2566 patch_prober.go:28] interesting pod/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Sep 11 14:29:07.603522 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:07.603495 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" podUID="a17db22a-8bd6-4571-a575-1b160cf26ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Sep 11 14:29:08.603889 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:08.603444 2566 patch_prober.go:28] interesting pod/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Sep 11 14:29:08.603889 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:08.603515 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" podUID="a17db22a-8bd6-4571-a575-1b160cf26ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Sep 11 14:29:09.603817 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:09.603774 2566 patch_prober.go:28] interesting pod/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Sep 11 14:29:09.604153 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:09.604122 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" podUID="a17db22a-8bd6-4571-a575-1b160cf26ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Sep 11 14:29:10.603074 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:10.603029 2566 patch_prober.go:28] interesting pod/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Sep 11 14:29:10.603312 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:10.603132 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" podUID="a17db22a-8bd6-4571-a575-1b160cf26ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Sep 11 14:29:11.602908 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:11.602872 2566 patch_prober.go:28] interesting pod/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Sep 11 14:29:11.603417 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:11.602957 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" podUID="a17db22a-8bd6-4571-a575-1b160cf26ff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Sep 11 14:29:12.613915 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:12.613878 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:29:12.614356 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:12.614283 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:29:12.615178 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:12.615162 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:29:21.965143 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:21.965079 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-rjfcb_417e7a61-fc2b-4125-97cf-53ef51e93982/payload-processing/0.log" Sep 11 14:29:58.176966 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:58.176934 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-rjfcb_417e7a61-fc2b-4125-97cf-53ef51e93982/payload-processing/0.log" Sep 11 14:29:59.123436 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.123400 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7"] Sep 11 14:29:59.127194 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.127177 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.131242 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.131220 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-e83e82-openshift-default-dockercfg-7wkf8\"" Sep 11 14:29:59.165821 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.165788 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-data\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.165983 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.165835 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.165983 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.165872 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-envoy\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.165983 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.165897 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-workload-certs\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.165983 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.165920 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-credential-socket\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.166183 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.165999 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-token\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.166183 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.166111 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-podinfo\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.166183 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.166151 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg8sz\" (UniqueName: \"kubernetes.io/projected/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-kube-api-access-gg8sz\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.166308 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.166202 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-workload-socket\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.255467 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.255438 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7"] Sep 11 14:29:59.267382 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.267356 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-workload-socket\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.267587 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.267428 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-data\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.267587 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.267475 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.267587 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.267499 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-envoy\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.267587 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.267526 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-workload-certs\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.267587 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.267552 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-credential-socket\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.267587 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.267578 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-token\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.267865 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.267657 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-podinfo\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.268145 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.268074 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg8sz\" (UniqueName: \"kubernetes.io/projected/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-kube-api-access-gg8sz\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.268679 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.268654 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-workload-socket\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.268865 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.268768 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-data\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.268865 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.268846 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-workload-certs\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.268974 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.268927 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-credential-socket\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.269063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.269039 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.270981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.270956 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-podinfo\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.271207 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.271188 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-envoy\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.285557 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.285529 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-token\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.304239 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.301811 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg8sz\" (UniqueName: \"kubernetes.io/projected/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-kube-api-access-gg8sz\") pod \"e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.438128 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.438002 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:29:59.739622 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.739594 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7"] Sep 11 14:29:59.741134 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:29:59.741066 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdfd40a0_7939_4f88_aca1_d10e2d6d10c0.slice/crio-c11ce8303fd997e0f8661cc18341c740c9667deaa90b5e84af8d498c6d71cf55 WatchSource:0}: Error finding container c11ce8303fd997e0f8661cc18341c740c9667deaa90b5e84af8d498c6d71cf55: Status 404 returned error can't find the container with id c11ce8303fd997e0f8661cc18341c740c9667deaa90b5e84af8d498c6d71cf55 Sep 11 14:29:59.743732 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.743698 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:29:59.743839 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.743781 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:29:59.743839 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.743825 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:29:59.964427 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.964391 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" event={"ID":"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0","Type":"ContainerStarted","Data":"ed0ba7c4ebe0ef4d2cf49b18868adb1a13f0c6a433a3967f5ce63329ef9c733a"} Sep 11 14:29:59.964625 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.964445 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" event={"ID":"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0","Type":"ContainerStarted","Data":"c11ce8303fd997e0f8661cc18341c740c9667deaa90b5e84af8d498c6d71cf55"} Sep 11 14:29:59.994349 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:29:59.994248 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" podStartSLOduration=0.994231947 podStartE2EDuration="994.231947ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:29:59.992421317 +0000 UTC m=+1940.397402446" watchObservedRunningTime="2026-09-11 14:29:59.994231947 +0000 UTC m=+1940.399213074" Sep 11 14:30:00.199045 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.199010 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-031a56-29818950-k5rnw"] Sep 11 14:30:00.202937 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.202915 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-30989f-29818950-xpx29"] Sep 11 14:30:00.203135 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.203040 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-031a56-29818950-k5rnw" Sep 11 14:30:00.206414 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.206386 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-30989f-29818950-xpx29" Sep 11 14:30:00.206529 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.206512 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:30:00.213337 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.213317 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1e2e7d-29818950-gw25w"] Sep 11 14:30:00.215383 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.215366 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-031a56-29818950-k5rnw" Sep 11 14:30:00.216718 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.216705 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1e2e7d-29818950-gw25w" Sep 11 14:30:00.225043 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.225015 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-30989f-29818950-xpx29" Sep 11 14:30:00.226163 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.226137 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-031a56-29818950-k5rnw"] Sep 11 14:30:00.231266 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.229704 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-30989f-29818950-xpx29"] Sep 11 14:30:00.233784 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.233762 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1e2e7d-29818950-gw25w" Sep 11 14:30:00.248021 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.247988 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1e2e7d-29818950-gw25w"] Sep 11 14:30:00.438744 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.438708 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:30:00.444313 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.444286 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:30:00.535778 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:30:00.535747 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6f820e_97ab_4b22_8a85_560283f275cc.slice/crio-3169a6a69a5eee58ba40a6770bb169fc811e67ca0a78aab53138b830bcbc292d WatchSource:0}: Error finding container 3169a6a69a5eee58ba40a6770bb169fc811e67ca0a78aab53138b830bcbc292d: Status 404 returned error can't find the container with id 3169a6a69a5eee58ba40a6770bb169fc811e67ca0a78aab53138b830bcbc292d Sep 11 14:30:00.537118 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.537079 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-031a56-29818950-k5rnw"] Sep 11 14:30:00.561393 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:30:00.561369 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8200f1aa_560a_474e_8e08_a21c127b2a8d.slice/crio-48cccb2a78b9bed1307fe9b080eb516e4835476316662fe8e06eb971930fc9d2 WatchSource:0}: Error finding container 48cccb2a78b9bed1307fe9b080eb516e4835476316662fe8e06eb971930fc9d2: Status 404 returned error can't find the container with id 48cccb2a78b9bed1307fe9b080eb516e4835476316662fe8e06eb971930fc9d2 Sep 11 14:30:00.561797 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.561774 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1e2e7d-29818950-gw25w"] Sep 11 14:30:00.568152 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.568131 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-30989f-29818950-xpx29"] Sep 11 14:30:00.568864 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:30:00.568844 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0c9101_2343_4a85_88bd_d1a6e965d28b.slice/crio-561dfd45f31ac7404d5497ed209a0af612ed8060f47e3c78bc4401739f6f2007 WatchSource:0}: Error finding container 561dfd45f31ac7404d5497ed209a0af612ed8060f47e3c78bc4401739f6f2007: Status 404 returned error can't find the container with id 561dfd45f31ac7404d5497ed209a0af612ed8060f47e3c78bc4401739f6f2007 Sep 11 14:30:00.972203 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.972164 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-031a56-29818950-k5rnw" event={"ID":"6d6f820e-97ab-4b22-8a85-560283f275cc","Type":"ContainerStarted","Data":"945597ac1ef3d9047cc8b3258618dda8cc13a10a2290ff1884924d295d786997"} Sep 11 14:30:00.972399 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.972255 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-031a56-29818950-k5rnw" event={"ID":"6d6f820e-97ab-4b22-8a85-560283f275cc","Type":"ContainerStarted","Data":"3169a6a69a5eee58ba40a6770bb169fc811e67ca0a78aab53138b830bcbc292d"} Sep 11 14:30:00.973512 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.973486 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-30989f-29818950-xpx29" event={"ID":"4f0c9101-2343-4a85-88bd-d1a6e965d28b","Type":"ContainerStarted","Data":"561dfd45f31ac7404d5497ed209a0af612ed8060f47e3c78bc4401739f6f2007"} Sep 11 14:30:00.974697 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.974677 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1e2e7d-29818950-gw25w" event={"ID":"8200f1aa-560a-474e-8e08-a21c127b2a8d","Type":"ContainerStarted","Data":"48cccb2a78b9bed1307fe9b080eb516e4835476316662fe8e06eb971930fc9d2"} Sep 11 14:30:00.974976 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.974960 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:30:00.975957 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.975939 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:30:00.997348 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:00.997300 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-031a56-29818950-k5rnw" podStartSLOduration=0.997283805 podStartE2EDuration="997.283805ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:00.997214078 +0000 UTC m=+1941.402195206" watchObservedRunningTime="2026-09-11 14:30:00.997283805 +0000 UTC m=+1941.402264934" Sep 11 14:30:01.982191 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:01.982156 2566 generic.go:356] "Generic (PLEG): container finished" podID="4f0c9101-2343-4a85-88bd-d1a6e965d28b" containerID="11f718dc8d448a341e259a8463d562c1cb581bbf8354a02c5bc80b2ff723b970" exitCode=0 Sep 11 14:30:01.982651 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:01.982250 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-30989f-29818950-xpx29" event={"ID":"4f0c9101-2343-4a85-88bd-d1a6e965d28b","Type":"ContainerDied","Data":"11f718dc8d448a341e259a8463d562c1cb581bbf8354a02c5bc80b2ff723b970"} Sep 11 14:30:01.983972 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:01.983951 2566 generic.go:356] "Generic (PLEG): container finished" podID="8200f1aa-560a-474e-8e08-a21c127b2a8d" containerID="313e6e2ee9c9c95325126b35bbfa7ed5fcc47cb4bb2dda8f967306d4cde15b25" exitCode=0 Sep 11 14:30:01.984095 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:01.984032 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1e2e7d-29818950-gw25w" event={"ID":"8200f1aa-560a-474e-8e08-a21c127b2a8d","Type":"ContainerDied","Data":"313e6e2ee9c9c95325126b35bbfa7ed5fcc47cb4bb2dda8f967306d4cde15b25"} Sep 11 14:30:01.985729 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:01.985706 2566 generic.go:356] "Generic (PLEG): container finished" podID="6d6f820e-97ab-4b22-8a85-560283f275cc" containerID="945597ac1ef3d9047cc8b3258618dda8cc13a10a2290ff1884924d295d786997" exitCode=0 Sep 11 14:30:01.985844 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:01.985728 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-031a56-29818950-k5rnw" event={"ID":"6d6f820e-97ab-4b22-8a85-560283f275cc","Type":"ContainerDied","Data":"945597ac1ef3d9047cc8b3258618dda8cc13a10a2290ff1884924d295d786997"} Sep 11 14:30:03.157668 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:03.157634 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-031a56-29818950-k5rnw" Sep 11 14:30:03.193737 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:03.193714 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-30989f-29818950-xpx29" Sep 11 14:30:03.197407 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:03.197376 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1e2e7d-29818950-gw25w" Sep 11 14:30:04.000716 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:04.000689 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-30989f-29818950-xpx29" Sep 11 14:30:04.000716 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:04.000703 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-30989f-29818950-xpx29" event={"ID":"4f0c9101-2343-4a85-88bd-d1a6e965d28b","Type":"ContainerDied","Data":"561dfd45f31ac7404d5497ed209a0af612ed8060f47e3c78bc4401739f6f2007"} Sep 11 14:30:04.000988 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:04.000757 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="561dfd45f31ac7404d5497ed209a0af612ed8060f47e3c78bc4401739f6f2007" Sep 11 14:30:04.002219 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:04.002191 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1e2e7d-29818950-gw25w" Sep 11 14:30:04.002425 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:04.002389 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1e2e7d-29818950-gw25w" event={"ID":"8200f1aa-560a-474e-8e08-a21c127b2a8d","Type":"ContainerDied","Data":"48cccb2a78b9bed1307fe9b080eb516e4835476316662fe8e06eb971930fc9d2"} Sep 11 14:30:04.002567 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:04.002443 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48cccb2a78b9bed1307fe9b080eb516e4835476316662fe8e06eb971930fc9d2" Sep 11 14:30:04.009182 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:04.009158 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-031a56-29818950-k5rnw" event={"ID":"6d6f820e-97ab-4b22-8a85-560283f275cc","Type":"ContainerDied","Data":"3169a6a69a5eee58ba40a6770bb169fc811e67ca0a78aab53138b830bcbc292d"} Sep 11 14:30:04.009298 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:04.009185 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-031a56-29818950-k5rnw" Sep 11 14:30:04.009298 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:04.009202 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3169a6a69a5eee58ba40a6770bb169fc811e67ca0a78aab53138b830bcbc292d" Sep 11 14:30:23.145001 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:23.144910 2566 generic.go:356] "Generic (PLEG): container finished" podID="bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" containerID="ed0ba7c4ebe0ef4d2cf49b18868adb1a13f0c6a433a3967f5ce63329ef9c733a" exitCode=0 Sep 11 14:30:23.145001 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:23.144969 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" event={"ID":"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0","Type":"ContainerDied","Data":"ed0ba7c4ebe0ef4d2cf49b18868adb1a13f0c6a433a3967f5ce63329ef9c733a"} Sep 11 14:30:23.145586 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:23.145544 2566 scope.go:117] "RemoveContainer" containerID="ed0ba7c4ebe0ef4d2cf49b18868adb1a13f0c6a433a3967f5ce63329ef9c733a" Sep 11 14:30:23.146756 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:23.146727 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:30:23.146814 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:23.146803 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:30:23.146857 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:23.146834 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:30:24.153234 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:24.153197 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" event={"ID":"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0","Type":"ContainerStarted","Data":"2eeca4f04e8e27dd2731a9f2d878471d0a8e8b4a3bfa3703047d59d8bacbcdae"} Sep 11 14:30:24.438566 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:24.438463 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:30:24.438734 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:24.438691 2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Sep 11 14:30:24.438800 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:24.438747 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" podUID="bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Sep 11 14:30:25.439284 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:25.439250 2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Sep 11 14:30:25.439689 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:25.439312 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" podUID="bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Sep 11 14:30:26.438654 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:26.438616 2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Sep 11 14:30:26.438840 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:26.438672 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" podUID="bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Sep 11 14:30:27.438475 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:27.438438 2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Sep 11 14:30:27.438897 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:27.438496 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" podUID="bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Sep 11 14:30:28.438780 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:28.438737 2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Sep 11 14:30:28.439301 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:28.438799 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" podUID="bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Sep 11 14:30:29.443208 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:29.443178 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:30:29.443654 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:29.443577 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:30:29.444154 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:29.444137 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:30:33.107857 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.107777 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss"] Sep 11 14:30:33.108274 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.108246 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8200f1aa-560a-474e-8e08-a21c127b2a8d" containerName="cleanup" Sep 11 14:30:33.108274 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.108259 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8200f1aa-560a-474e-8e08-a21c127b2a8d" containerName="cleanup" Sep 11 14:30:33.108358 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.108285 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d6f820e-97ab-4b22-8a85-560283f275cc" containerName="cleanup" Sep 11 14:30:33.108358 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.108290 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d6f820e-97ab-4b22-8a85-560283f275cc" containerName="cleanup" Sep 11 14:30:33.108358 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.108299 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f0c9101-2343-4a85-88bd-d1a6e965d28b" containerName="cleanup" Sep 11 14:30:33.108358 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.108306 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f0c9101-2343-4a85-88bd-d1a6e965d28b" containerName="cleanup" Sep 11 14:30:33.108510 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.108370 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d6f820e-97ab-4b22-8a85-560283f275cc" containerName="cleanup" Sep 11 14:30:33.108510 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.108378 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f0c9101-2343-4a85-88bd-d1a6e965d28b" containerName="cleanup" Sep 11 14:30:33.108510 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.108385 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8200f1aa-560a-474e-8e08-a21c127b2a8d" containerName="cleanup" Sep 11 14:30:33.112578 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.112558 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss" Sep 11 14:30:33.115862 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.115841 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:30:33.134181 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.134146 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b08a364e-9478-4f34-9a3e-eda7b336eea0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss\" (UID: \"b08a364e-9478-4f34-9a3e-eda7b336eea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss" Sep 11 14:30:33.137512 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.137490 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss"] Sep 11 14:30:33.235282 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.235248 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b08a364e-9478-4f34-9a3e-eda7b336eea0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss\" (UID: \"b08a364e-9478-4f34-9a3e-eda7b336eea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss" Sep 11 14:30:33.235899 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.235880 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b08a364e-9478-4f34-9a3e-eda7b336eea0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss\" (UID: \"b08a364e-9478-4f34-9a3e-eda7b336eea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss" Sep 11 14:30:33.422794 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.422704 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss" Sep 11 14:30:33.709342 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:33.709310 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss"] Sep 11 14:30:33.709817 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:30:33.709788 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08a364e_9478_4f34_9a3e_eda7b336eea0.slice/crio-5dd9febda4ccb0b835a0f5ad8c80834c34b5cf4eb83e2161a8e3bb173288b3a0 WatchSource:0}: Error finding container 5dd9febda4ccb0b835a0f5ad8c80834c34b5cf4eb83e2161a8e3bb173288b3a0: Status 404 returned error can't find the container with id 5dd9febda4ccb0b835a0f5ad8c80834c34b5cf4eb83e2161a8e3bb173288b3a0 Sep 11 14:30:34.226004 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:34.225957 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss" event={"ID":"b08a364e-9478-4f34-9a3e-eda7b336eea0","Type":"ContainerStarted","Data":"3358acf7ebd0faaa7ea26b089c42cd8070f04bff3e5c6d875a18d6c4f51144ef"} Sep 11 14:30:34.226536 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:34.226024 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss" event={"ID":"b08a364e-9478-4f34-9a3e-eda7b336eea0","Type":"ContainerStarted","Data":"5dd9febda4ccb0b835a0f5ad8c80834c34b5cf4eb83e2161a8e3bb173288b3a0"} Sep 11 14:30:34.252498 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:34.252448 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss" podStartSLOduration=1.252432041 podStartE2EDuration="1.252432041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:34.249762682 +0000 UTC m=+1974.654743810" watchObservedRunningTime="2026-09-11 14:30:34.252432041 +0000 UTC m=+1974.657413171" Sep 11 14:30:35.233557 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:35.233519 2566 generic.go:356] "Generic (PLEG): container finished" podID="b08a364e-9478-4f34-9a3e-eda7b336eea0" containerID="3358acf7ebd0faaa7ea26b089c42cd8070f04bff3e5c6d875a18d6c4f51144ef" exitCode=0 Sep 11 14:30:35.233941 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:35.233601 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss" event={"ID":"b08a364e-9478-4f34-9a3e-eda7b336eea0","Type":"ContainerDied","Data":"3358acf7ebd0faaa7ea26b089c42cd8070f04bff3e5c6d875a18d6c4f51144ef"} Sep 11 14:30:36.383214 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:36.383190 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss" Sep 11 14:30:36.469418 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:36.469383 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b08a364e-9478-4f34-9a3e-eda7b336eea0-maas-api-service-ca\") pod \"b08a364e-9478-4f34-9a3e-eda7b336eea0\" (UID: \"b08a364e-9478-4f34-9a3e-eda7b336eea0\") " Sep 11 14:30:36.469738 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:36.469715 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08a364e-9478-4f34-9a3e-eda7b336eea0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b08a364e-9478-4f34-9a3e-eda7b336eea0" (UID: "b08a364e-9478-4f34-9a3e-eda7b336eea0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:30:36.469845 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:36.469827 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b08a364e-9478-4f34-9a3e-eda7b336eea0-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:30:37.247398 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:37.247361 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss" Sep 11 14:30:37.247398 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:37.247383 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss" event={"ID":"b08a364e-9478-4f34-9a3e-eda7b336eea0","Type":"ContainerDied","Data":"5dd9febda4ccb0b835a0f5ad8c80834c34b5cf4eb83e2161a8e3bb173288b3a0"} Sep 11 14:30:37.247629 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:37.247429 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dd9febda4ccb0b835a0f5ad8c80834c34b5cf4eb83e2161a8e3bb173288b3a0" Sep 11 14:30:45.074415 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:45.074374 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss"] Sep 11 14:30:45.085981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:45.085945 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e83e82-xmtss"] Sep 11 14:30:46.098675 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:46.098638 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b08a364e-9478-4f34-9a3e-eda7b336eea0" path="/var/lib/kubelet/pods/b08a364e-9478-4f34-9a3e-eda7b336eea0/volumes" Sep 11 14:30:52.170779 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:52.170743 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7"] Sep 11 14:30:52.171190 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:52.170992 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" podUID="bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" containerName="istio-proxy" containerID="cri-o://2eeca4f04e8e27dd2731a9f2d878471d0a8e8b4a3bfa3703047d59d8bacbcdae" gracePeriod=30 Sep 11 14:30:56.075186 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:56.075154 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp"] Sep 11 14:30:56.075593 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:56.075580 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08a364e-9478-4f34-9a3e-eda7b336eea0" containerName="revoke-keys" Sep 11 14:30:56.075638 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:56.075594 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08a364e-9478-4f34-9a3e-eda7b336eea0" containerName="revoke-keys" Sep 11 14:30:56.075674 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:56.075662 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b08a364e-9478-4f34-9a3e-eda7b336eea0" containerName="revoke-keys" Sep 11 14:30:56.078893 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:56.078877 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp" Sep 11 14:30:56.081762 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:56.081740 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:30:56.093512 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:56.093486 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp"] Sep 11 14:30:56.172394 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:56.172361 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba003a2-3bc9-4065-8914-dc6462dc0da5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp\" (UID: \"7ba003a2-3bc9-4065-8914-dc6462dc0da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp" Sep 11 14:30:56.273608 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:56.273567 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba003a2-3bc9-4065-8914-dc6462dc0da5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp\" (UID: \"7ba003a2-3bc9-4065-8914-dc6462dc0da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp" Sep 11 14:30:56.274219 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:56.274197 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba003a2-3bc9-4065-8914-dc6462dc0da5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp\" (UID: \"7ba003a2-3bc9-4065-8914-dc6462dc0da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp" Sep 11 14:30:56.395624 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:56.395541 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp" Sep 11 14:30:56.691353 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:56.691316 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp"] Sep 11 14:30:56.692356 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:30:56.692333 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba003a2_3bc9_4065_8914_dc6462dc0da5.slice/crio-a824f357ea9801cf3ff49e965fc828aacc3b23e7594418ebb913987216ef59cd WatchSource:0}: Error finding container a824f357ea9801cf3ff49e965fc828aacc3b23e7594418ebb913987216ef59cd: Status 404 returned error can't find the container with id a824f357ea9801cf3ff49e965fc828aacc3b23e7594418ebb913987216ef59cd Sep 11 14:30:57.388896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.388851 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp" event={"ID":"7ba003a2-3bc9-4065-8914-dc6462dc0da5","Type":"ContainerStarted","Data":"a2580b4897b898514923637c76d6e1622c45f6da228c3c148165c3dfb77d1ba8"} Sep 11 14:30:57.389386 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.388918 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp" event={"ID":"7ba003a2-3bc9-4065-8914-dc6462dc0da5","Type":"ContainerStarted","Data":"a824f357ea9801cf3ff49e965fc828aacc3b23e7594418ebb913987216ef59cd"} Sep 11 14:30:57.390867 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.390842 2566 generic.go:356] "Generic (PLEG): container finished" podID="bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" containerID="2eeca4f04e8e27dd2731a9f2d878471d0a8e8b4a3bfa3703047d59d8bacbcdae" exitCode=0 Sep 11 14:30:57.390970 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.390912 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" event={"ID":"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0","Type":"ContainerDied","Data":"2eeca4f04e8e27dd2731a9f2d878471d0a8e8b4a3bfa3703047d59d8bacbcdae"} Sep 11 14:30:57.391445 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.390968 2566 scope.go:117] "RemoveContainer" containerID="ed0ba7c4ebe0ef4d2cf49b18868adb1a13f0c6a433a3967f5ce63329ef9c733a" Sep 11 14:30:57.412843 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.412781 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp" podStartSLOduration=1.412765092 podStartE2EDuration="1.412765092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:57.409885831 +0000 UTC m=+1997.814866959" watchObservedRunningTime="2026-09-11 14:30:57.412765092 +0000 UTC m=+1997.817746219" Sep 11 14:30:57.487892 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.487871 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:30:57.585886 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.585792 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-token\") pod \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " Sep 11 14:30:57.585886 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.585866 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-podinfo\") pod \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " Sep 11 14:30:57.586153 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.585896 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istiod-ca-cert\") pod \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " Sep 11 14:30:57.586153 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.585924 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-workload-socket\") pod \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " Sep 11 14:30:57.586153 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.585974 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-workload-certs\") pod \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " Sep 11 14:30:57.586153 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.586021 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-credential-socket\") pod \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " Sep 11 14:30:57.586153 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.586044 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-data\") pod \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " Sep 11 14:30:57.586153 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.586124 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg8sz\" (UniqueName: \"kubernetes.io/projected/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-kube-api-access-gg8sz\") pod \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " Sep 11 14:30:57.586469 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.586159 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-envoy\") pod \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\" (UID: \"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0\") " Sep 11 14:30:57.586469 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.586292 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" (UID: "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:30:57.586469 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.586292 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" (UID: "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:30:57.586469 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.586316 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" (UID: "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:30:57.586469 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.586304 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" (UID: "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:30:57.586705 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.586508 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-data" (OuterVolumeSpecName: "istio-data") pod "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" (UID: "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:30:57.586705 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.586611 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:30:57.586705 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.586629 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:30:57.586705 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.586645 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:30:57.586705 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.586662 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:30:57.586705 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.586678 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:30:57.588524 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.588503 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-kube-api-access-gg8sz" (OuterVolumeSpecName: "kube-api-access-gg8sz") pod "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" (UID: "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0"). InnerVolumeSpecName "kube-api-access-gg8sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:30:57.588598 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.588563 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" (UID: "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:30:57.588647 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.588622 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" (UID: "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:30:57.588685 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.588639 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" (UID: "bdfd40a0-7939-4f88-aca1-d10e2d6d10c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:30:57.687549 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.687514 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:30:57.687549 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.687543 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:30:57.687549 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.687553 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg8sz\" (UniqueName: \"kubernetes.io/projected/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-kube-api-access-gg8sz\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:30:57.687549 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:57.687561 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:30:58.399477 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:58.399375 2566 generic.go:356] "Generic (PLEG): container finished" podID="7ba003a2-3bc9-4065-8914-dc6462dc0da5" containerID="a2580b4897b898514923637c76d6e1622c45f6da228c3c148165c3dfb77d1ba8" exitCode=0 Sep 11 14:30:58.399477 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:58.399414 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp" event={"ID":"7ba003a2-3bc9-4065-8914-dc6462dc0da5","Type":"ContainerDied","Data":"a2580b4897b898514923637c76d6e1622c45f6da228c3c148165c3dfb77d1ba8"} Sep 11 14:30:58.401047 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:58.401020 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" event={"ID":"bdfd40a0-7939-4f88-aca1-d10e2d6d10c0","Type":"ContainerDied","Data":"c11ce8303fd997e0f8661cc18341c740c9667deaa90b5e84af8d498c6d71cf55"} Sep 11 14:30:58.401199 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:58.401046 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7" Sep 11 14:30:58.401199 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:58.401076 2566 scope.go:117] "RemoveContainer" containerID="2eeca4f04e8e27dd2731a9f2d878471d0a8e8b4a3bfa3703047d59d8bacbcdae" Sep 11 14:30:58.464219 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:58.464188 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7"] Sep 11 14:30:58.469664 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:58.469636 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e83e82-openshift-default-756bf5d7db-gqpf7"] Sep 11 14:30:59.548283 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:59.548260 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp" Sep 11 14:30:59.707747 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:59.707666 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba003a2-3bc9-4065-8914-dc6462dc0da5-maas-api-service-ca\") pod \"7ba003a2-3bc9-4065-8914-dc6462dc0da5\" (UID: \"7ba003a2-3bc9-4065-8914-dc6462dc0da5\") " Sep 11 14:30:59.708136 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:59.708073 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba003a2-3bc9-4065-8914-dc6462dc0da5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ba003a2-3bc9-4065-8914-dc6462dc0da5" (UID: "7ba003a2-3bc9-4065-8914-dc6462dc0da5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:30:59.809293 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:30:59.809259 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba003a2-3bc9-4065-8914-dc6462dc0da5-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:31:00.098948 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:31:00.098915 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" path="/var/lib/kubelet/pods/bdfd40a0-7939-4f88-aca1-d10e2d6d10c0/volumes" Sep 11 14:31:00.417582 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:31:00.417495 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp" Sep 11 14:31:00.417748 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:31:00.417494 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp" event={"ID":"7ba003a2-3bc9-4065-8914-dc6462dc0da5","Type":"ContainerDied","Data":"a824f357ea9801cf3ff49e965fc828aacc3b23e7594418ebb913987216ef59cd"} Sep 11 14:31:00.417748 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:31:00.417630 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a824f357ea9801cf3ff49e965fc828aacc3b23e7594418ebb913987216ef59cd" Sep 11 14:31:11.986058 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:31:11.986022 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-30989f-29818950-xpx29"] Sep 11 14:31:11.989611 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:31:11.989584 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-30989f-29818950-xpx29"] Sep 11 14:31:12.100299 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:31:12.100262 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f0c9101-2343-4a85-88bd-d1a6e965d28b" path="/var/lib/kubelet/pods/4f0c9101-2343-4a85-88bd-d1a6e965d28b/volumes" Sep 11 14:31:12.382936 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:31:12.382854 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp"] Sep 11 14:31:12.387418 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:31:12.387390 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-30989f-qjxfp"] Sep 11 14:31:14.098066 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:31:14.098033 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ba003a2-3bc9-4065-8914-dc6462dc0da5" path="/var/lib/kubelet/pods/7ba003a2-3bc9-4065-8914-dc6462dc0da5/volumes" Sep 11 14:31:40.625634 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:31:40.625601 2566 scope.go:117] "RemoveContainer" containerID="68fa4273328d7ffc00df92189b8c5024aa12f150432a3492348379ad2006a693" Sep 11 14:32:01.609601 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:01.609511 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7"] Sep 11 14:32:01.610032 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:01.609802 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" podUID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" containerName="istio-proxy" containerID="cri-o://91cf779aff8c26e26008995b9ba9fa7674ee5dd74529eaf1b26d4d8f5bb944b8" gracePeriod=30 Sep 11 14:32:01.981572 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:01.981536 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q"] Sep 11 14:32:01.981956 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:01.981942 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ba003a2-3bc9-4065-8914-dc6462dc0da5" containerName="revoke-keys" Sep 11 14:32:01.981956 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:01.981956 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ba003a2-3bc9-4065-8914-dc6462dc0da5" containerName="revoke-keys" Sep 11 14:32:01.982112 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:01.981972 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" containerName="istio-proxy" Sep 11 14:32:01.982112 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:01.981977 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" containerName="istio-proxy" Sep 11 14:32:01.982112 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:01.982053 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" containerName="istio-proxy" Sep 11 14:32:01.982112 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:01.982061 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ba003a2-3bc9-4065-8914-dc6462dc0da5" containerName="revoke-keys" Sep 11 14:32:01.982112 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:01.982068 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" containerName="istio-proxy" Sep 11 14:32:01.985219 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:01.985202 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q" Sep 11 14:32:01.989099 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:01.989065 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrfn2\"" Sep 11 14:32:01.994655 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:01.994636 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q"] Sep 11 14:32:02.114407 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:02.114371 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ff19ef-39d2-44f3-90f4-178a59eee550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q\" (UID: \"63ff19ef-39d2-44f3-90f4-178a59eee550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q" Sep 11 14:32:02.215455 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:02.215424 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ff19ef-39d2-44f3-90f4-178a59eee550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q\" (UID: \"63ff19ef-39d2-44f3-90f4-178a59eee550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q" Sep 11 14:32:02.216062 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:02.216039 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ff19ef-39d2-44f3-90f4-178a59eee550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q\" (UID: \"63ff19ef-39d2-44f3-90f4-178a59eee550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q" Sep 11 14:32:02.296722 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:02.296636 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q" Sep 11 14:32:02.574032 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:02.574001 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q"] Sep 11 14:32:02.575408 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:32:02.575386 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ff19ef_39d2_44f3_90f4_178a59eee550.slice/crio-04a824c1080ed7d7523853b200f50e16b8a141caa2b05008647a139a869caf06 WatchSource:0}: Error finding container 04a824c1080ed7d7523853b200f50e16b8a141caa2b05008647a139a869caf06: Status 404 returned error can't find the container with id 04a824c1080ed7d7523853b200f50e16b8a141caa2b05008647a139a869caf06 Sep 11 14:32:02.818940 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:02.818913 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q" event={"ID":"63ff19ef-39d2-44f3-90f4-178a59eee550","Type":"ContainerStarted","Data":"04a824c1080ed7d7523853b200f50e16b8a141caa2b05008647a139a869caf06"} Sep 11 14:32:03.827036 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:03.827001 2566 generic.go:356] "Generic (PLEG): container finished" podID="63ff19ef-39d2-44f3-90f4-178a59eee550" containerID="9f9b6127e17a3d82833936ffc74f69396ea22f9c607964088b8813e7db8261d5" exitCode=0 Sep 11 14:32:03.827503 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:03.827105 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q" event={"ID":"63ff19ef-39d2-44f3-90f4-178a59eee550","Type":"ContainerDied","Data":"9f9b6127e17a3d82833936ffc74f69396ea22f9c607964088b8813e7db8261d5"} Sep 11 14:32:04.974611 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:04.974584 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q" Sep 11 14:32:05.043416 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:05.043375 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ff19ef-39d2-44f3-90f4-178a59eee550-maas-api-service-ca\") pod \"63ff19ef-39d2-44f3-90f4-178a59eee550\" (UID: \"63ff19ef-39d2-44f3-90f4-178a59eee550\") " Sep 11 14:32:05.043747 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:05.043723 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ff19ef-39d2-44f3-90f4-178a59eee550-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63ff19ef-39d2-44f3-90f4-178a59eee550" (UID: "63ff19ef-39d2-44f3-90f4-178a59eee550"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:32:05.145254 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:05.145156 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ff19ef-39d2-44f3-90f4-178a59eee550-maas-api-service-ca\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:05.841493 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:05.841463 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q" Sep 11 14:32:05.841698 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:05.841464 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q" event={"ID":"63ff19ef-39d2-44f3-90f4-178a59eee550","Type":"ContainerDied","Data":"04a824c1080ed7d7523853b200f50e16b8a141caa2b05008647a139a869caf06"} Sep 11 14:32:05.841698 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:05.841584 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04a824c1080ed7d7523853b200f50e16b8a141caa2b05008647a139a869caf06" Sep 11 14:32:06.851001 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.850975 2566 generic.go:356] "Generic (PLEG): container finished" podID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" containerID="91cf779aff8c26e26008995b9ba9fa7674ee5dd74529eaf1b26d4d8f5bb944b8" exitCode=0 Sep 11 14:32:06.851388 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.851042 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" event={"ID":"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f","Type":"ContainerDied","Data":"91cf779aff8c26e26008995b9ba9fa7674ee5dd74529eaf1b26d4d8f5bb944b8"} Sep 11 14:32:06.851388 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.851133 2566 scope.go:117] "RemoveContainer" containerID="a2a7acb0e73dce2481fc470da080ca9674305f60b90d87879c4a6d2a133a6027" Sep 11 14:32:06.873939 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.873919 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:32:06.965318 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.965283 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-workload-socket\") pod \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " Sep 11 14:32:06.965514 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.965345 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-podinfo\") pod \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " Sep 11 14:32:06.965514 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.965374 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-token\") pod \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " Sep 11 14:32:06.965514 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.965440 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-envoy\") pod \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " Sep 11 14:32:06.965514 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.965464 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istiod-ca-cert\") pod \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " Sep 11 14:32:06.965514 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.965486 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-credential-socket\") pod \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " Sep 11 14:32:06.965514 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.965511 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-data\") pod \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " Sep 11 14:32:06.965832 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.965540 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4w4c\" (UniqueName: \"kubernetes.io/projected/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-kube-api-access-b4w4c\") pod \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " Sep 11 14:32:06.965832 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.965557 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" (UID: "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:06.965832 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.965640 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-workload-certs\") pod \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\" (UID: \"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f\") " Sep 11 14:32:06.965993 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.965860 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" (UID: "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:06.965993 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.965883 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" (UID: "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:32:06.965993 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.965911 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-data" (OuterVolumeSpecName: "istio-data") pod "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" (UID: "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:06.966178 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.966054 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" (UID: "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:06.966178 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.966154 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:06.966178 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.966176 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:06.966294 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.966190 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:06.966294 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.966203 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:06.967732 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.967703 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" (UID: "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:32:06.968137 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.968119 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-kube-api-access-b4w4c" (OuterVolumeSpecName: "kube-api-access-b4w4c") pod "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" (UID: "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f"). InnerVolumeSpecName "kube-api-access-b4w4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:06.968312 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.968286 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" (UID: "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:06.968412 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:06.968308 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-token" (OuterVolumeSpecName: "istio-token") pod "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" (UID: "6aa5f5af-9d1f-4095-96e5-0e64f52fde1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:07.066826 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:07.066790 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:07.066826 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:07.066821 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:07.066826 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:07.066831 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4w4c\" (UniqueName: \"kubernetes.io/projected/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-kube-api-access-b4w4c\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:07.067063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:07.066843 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:07.067063 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:07.066852 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:07.858896 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:07.858862 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" Sep 11 14:32:07.859380 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:07.858906 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7" event={"ID":"6aa5f5af-9d1f-4095-96e5-0e64f52fde1f","Type":"ContainerDied","Data":"8b583bfc631cd1bd2e550188c7fb9d793c42f8347fe94cb1aae547432e0e8868"} Sep 11 14:32:07.859380 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:07.858969 2566 scope.go:117] "RemoveContainer" containerID="91cf779aff8c26e26008995b9ba9fa7674ee5dd74529eaf1b26d4d8f5bb944b8" Sep 11 14:32:07.897666 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:07.897634 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7"] Sep 11 14:32:07.909028 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:07.909003 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-30989f-openshift-default-5d85d49d8-ftcq7"] Sep 11 14:32:08.098006 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:08.097964 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" path="/var/lib/kubelet/pods/6aa5f5af-9d1f-4095-96e5-0e64f52fde1f/volumes" Sep 11 14:32:13.030044 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:13.030005 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q"] Sep 11 14:32:13.036135 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:13.036105 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b7d522-wkn9q"] Sep 11 14:32:14.098406 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:14.098372 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63ff19ef-39d2-44f3-90f4-178a59eee550" path="/var/lib/kubelet/pods/63ff19ef-39d2-44f3-90f4-178a59eee550/volumes" Sep 11 14:32:20.175645 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:20.175607 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h"] Sep 11 14:32:20.176039 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:20.175816 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" podUID="a17db22a-8bd6-4571-a575-1b160cf26ff2" containerName="istio-proxy" containerID="cri-o://28250acfcf630c16e7e9d26e06567e97e7f49a9288388da4a424d08e3c0ee8e7" gracePeriod=30 Sep 11 14:32:25.450011 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.449946 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:32:25.539792 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.539760 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-workload-socket\") pod \"a17db22a-8bd6-4571-a575-1b160cf26ff2\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " Sep 11 14:32:25.539792 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.539797 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-token\") pod \"a17db22a-8bd6-4571-a575-1b160cf26ff2\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " Sep 11 14:32:25.540054 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.539820 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17db22a-8bd6-4571-a575-1b160cf26ff2-istiod-ca-cert\") pod \"a17db22a-8bd6-4571-a575-1b160cf26ff2\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " Sep 11 14:32:25.540054 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.539862 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-podinfo\") pod \"a17db22a-8bd6-4571-a575-1b160cf26ff2\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " Sep 11 14:32:25.540054 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.539915 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-data\") pod \"a17db22a-8bd6-4571-a575-1b160cf26ff2\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " Sep 11 14:32:25.540054 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.539932 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-workload-certs\") pod \"a17db22a-8bd6-4571-a575-1b160cf26ff2\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " Sep 11 14:32:25.540054 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.539966 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7glc\" (UniqueName: \"kubernetes.io/projected/a17db22a-8bd6-4571-a575-1b160cf26ff2-kube-api-access-b7glc\") pod \"a17db22a-8bd6-4571-a575-1b160cf26ff2\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " Sep 11 14:32:25.540054 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.540010 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-envoy\") pod \"a17db22a-8bd6-4571-a575-1b160cf26ff2\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " Sep 11 14:32:25.540419 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.540068 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-credential-socket\") pod \"a17db22a-8bd6-4571-a575-1b160cf26ff2\" (UID: \"a17db22a-8bd6-4571-a575-1b160cf26ff2\") " Sep 11 14:32:25.540419 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.540065 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a17db22a-8bd6-4571-a575-1b160cf26ff2" (UID: "a17db22a-8bd6-4571-a575-1b160cf26ff2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:25.540419 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.540301 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-data" (OuterVolumeSpecName: "istio-data") pod "a17db22a-8bd6-4571-a575-1b160cf26ff2" (UID: "a17db22a-8bd6-4571-a575-1b160cf26ff2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:25.540573 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.540447 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:25.540573 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.540435 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17db22a-8bd6-4571-a575-1b160cf26ff2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a17db22a-8bd6-4571-a575-1b160cf26ff2" (UID: "a17db22a-8bd6-4571-a575-1b160cf26ff2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:32:25.540573 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.540466 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:25.540573 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.540491 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a17db22a-8bd6-4571-a575-1b160cf26ff2" (UID: "a17db22a-8bd6-4571-a575-1b160cf26ff2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:25.540573 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.540532 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a17db22a-8bd6-4571-a575-1b160cf26ff2" (UID: "a17db22a-8bd6-4571-a575-1b160cf26ff2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:25.542358 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.542313 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a17db22a-8bd6-4571-a575-1b160cf26ff2" (UID: "a17db22a-8bd6-4571-a575-1b160cf26ff2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:32:25.542358 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.542347 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a17db22a-8bd6-4571-a575-1b160cf26ff2" (UID: "a17db22a-8bd6-4571-a575-1b160cf26ff2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:25.542564 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.542363 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-token" (OuterVolumeSpecName: "istio-token") pod "a17db22a-8bd6-4571-a575-1b160cf26ff2" (UID: "a17db22a-8bd6-4571-a575-1b160cf26ff2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:25.542564 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.542420 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17db22a-8bd6-4571-a575-1b160cf26ff2-kube-api-access-b7glc" (OuterVolumeSpecName: "kube-api-access-b7glc") pod "a17db22a-8bd6-4571-a575-1b160cf26ff2" (UID: "a17db22a-8bd6-4571-a575-1b160cf26ff2"). InnerVolumeSpecName "kube-api-access-b7glc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:25.640931 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.640893 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:25.640931 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.640924 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7glc\" (UniqueName: \"kubernetes.io/projected/a17db22a-8bd6-4571-a575-1b160cf26ff2-kube-api-access-b7glc\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:25.640931 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.640938 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:25.641233 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.640950 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17db22a-8bd6-4571-a575-1b160cf26ff2-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:25.641233 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.640962 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:25.641233 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.640974 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17db22a-8bd6-4571-a575-1b160cf26ff2-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:25.641233 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.640984 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17db22a-8bd6-4571-a575-1b160cf26ff2-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:25.991750 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.991712 2566 generic.go:356] "Generic (PLEG): container finished" podID="a17db22a-8bd6-4571-a575-1b160cf26ff2" containerID="28250acfcf630c16e7e9d26e06567e97e7f49a9288388da4a424d08e3c0ee8e7" exitCode=0 Sep 11 14:32:25.991927 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.991769 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" Sep 11 14:32:25.991927 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.991806 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" event={"ID":"a17db22a-8bd6-4571-a575-1b160cf26ff2","Type":"ContainerDied","Data":"28250acfcf630c16e7e9d26e06567e97e7f49a9288388da4a424d08e3c0ee8e7"} Sep 11 14:32:25.991927 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.991869 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h" event={"ID":"a17db22a-8bd6-4571-a575-1b160cf26ff2","Type":"ContainerDied","Data":"2b92be7ddf30d934f7e782703f585307a4ec8c3ae1943d1f46c0cc4ae6f517c6"} Sep 11 14:32:25.991927 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:25.991901 2566 scope.go:117] "RemoveContainer" containerID="28250acfcf630c16e7e9d26e06567e97e7f49a9288388da4a424d08e3c0ee8e7" Sep 11 14:32:26.002936 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:26.002913 2566 scope.go:117] "RemoveContainer" containerID="0db5ed0db24752dbb7a28a15290d0eec6a7691139876c50bea7acf129bf24631" Sep 11 14:32:26.011929 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:26.011910 2566 scope.go:117] "RemoveContainer" containerID="28250acfcf630c16e7e9d26e06567e97e7f49a9288388da4a424d08e3c0ee8e7" Sep 11 14:32:26.012211 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:32:26.012191 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28250acfcf630c16e7e9d26e06567e97e7f49a9288388da4a424d08e3c0ee8e7\": container with ID starting with 28250acfcf630c16e7e9d26e06567e97e7f49a9288388da4a424d08e3c0ee8e7 not found: ID does not exist" containerID="28250acfcf630c16e7e9d26e06567e97e7f49a9288388da4a424d08e3c0ee8e7" Sep 11 14:32:26.012265 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:26.012221 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28250acfcf630c16e7e9d26e06567e97e7f49a9288388da4a424d08e3c0ee8e7"} err="failed to get container status \"28250acfcf630c16e7e9d26e06567e97e7f49a9288388da4a424d08e3c0ee8e7\": rpc error: code = NotFound desc = could not find container \"28250acfcf630c16e7e9d26e06567e97e7f49a9288388da4a424d08e3c0ee8e7\": container with ID starting with 28250acfcf630c16e7e9d26e06567e97e7f49a9288388da4a424d08e3c0ee8e7 not found: ID does not exist" Sep 11 14:32:26.012265 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:26.012238 2566 scope.go:117] "RemoveContainer" containerID="0db5ed0db24752dbb7a28a15290d0eec6a7691139876c50bea7acf129bf24631" Sep 11 14:32:26.012507 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:32:26.012488 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db5ed0db24752dbb7a28a15290d0eec6a7691139876c50bea7acf129bf24631\": container with ID starting with 0db5ed0db24752dbb7a28a15290d0eec6a7691139876c50bea7acf129bf24631 not found: ID does not exist" containerID="0db5ed0db24752dbb7a28a15290d0eec6a7691139876c50bea7acf129bf24631" Sep 11 14:32:26.012548 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:26.012514 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db5ed0db24752dbb7a28a15290d0eec6a7691139876c50bea7acf129bf24631"} err="failed to get container status \"0db5ed0db24752dbb7a28a15290d0eec6a7691139876c50bea7acf129bf24631\": rpc error: code = NotFound desc = could not find container \"0db5ed0db24752dbb7a28a15290d0eec6a7691139876c50bea7acf129bf24631\": container with ID starting with 0db5ed0db24752dbb7a28a15290d0eec6a7691139876c50bea7acf129bf24631 not found: ID does not exist" Sep 11 14:32:26.017776 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:26.017753 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h"] Sep 11 14:32:26.025306 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:26.025286 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b7d522-openshift-default-5d649d4f68-c5w8h"] Sep 11 14:32:26.097529 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:26.097498 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a17db22a-8bd6-4571-a575-1b160cf26ff2" path="/var/lib/kubelet/pods/a17db22a-8bd6-4571-a575-1b160cf26ff2/volumes" Sep 11 14:32:31.434959 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:31.434922 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1e2e7d-29818950-gw25w"] Sep 11 14:32:31.439425 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:31.439402 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1e2e7d-29818950-gw25w"] Sep 11 14:32:32.098168 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:32.098129 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8200f1aa-560a-474e-8e08-a21c127b2a8d" path="/var/lib/kubelet/pods/8200f1aa-560a-474e-8e08-a21c127b2a8d/volumes" Sep 11 14:32:39.505816 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:39.505779 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw"] Sep 11 14:32:39.506273 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:39.506020 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerName="istio-proxy" containerID="cri-o://30c1f82f596d356199794ba8bd99e298dc2ad4ba0f2fe2a9db5fde11dac75044" gracePeriod=30 Sep 11 14:32:40.283534 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:40.283498 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:32:40.288383 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:40.288352 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:32:40.659226 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:40.659132 2566 scope.go:117] "RemoveContainer" containerID="cdb97a1a62bec6be60deae8af5022a9156cbb274596bb36482d485cc20a38bcc" Sep 11 14:32:40.668434 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:40.668415 2566 scope.go:117] "RemoveContainer" containerID="ba77bfdc6249c6b979e7cf329c2085703be87af8c9dc64522b7b96f7b0d4dfb5" Sep 11 14:32:40.676941 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:40.676920 2566 scope.go:117] "RemoveContainer" containerID="f08354a23a824da92e61452ae03ebdf70cfa6e457588467af4dfda5412f39211" Sep 11 14:32:44.759345 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.759316 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:32:44.831758 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.831654 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qk4s\" (UniqueName: \"kubernetes.io/projected/8849f8ab-2b78-439c-911c-323c8c21c8b3-kube-api-access-8qk4s\") pod \"8849f8ab-2b78-439c-911c-323c8c21c8b3\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " Sep 11 14:32:44.831758 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.831695 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-data\") pod \"8849f8ab-2b78-439c-911c-323c8c21c8b3\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " Sep 11 14:32:44.831758 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.831738 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-workload-socket\") pod \"8849f8ab-2b78-439c-911c-323c8c21c8b3\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " Sep 11 14:32:44.832039 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.831771 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8849f8ab-2b78-439c-911c-323c8c21c8b3-istiod-ca-cert\") pod \"8849f8ab-2b78-439c-911c-323c8c21c8b3\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " Sep 11 14:32:44.832039 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.831800 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-token\") pod \"8849f8ab-2b78-439c-911c-323c8c21c8b3\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " Sep 11 14:32:44.832039 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.831851 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-workload-certs\") pod \"8849f8ab-2b78-439c-911c-323c8c21c8b3\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " Sep 11 14:32:44.832039 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.831900 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-credential-socket\") pod \"8849f8ab-2b78-439c-911c-323c8c21c8b3\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " Sep 11 14:32:44.832039 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.831945 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-podinfo\") pod \"8849f8ab-2b78-439c-911c-323c8c21c8b3\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " Sep 11 14:32:44.832039 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.831969 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-envoy\") pod \"8849f8ab-2b78-439c-911c-323c8c21c8b3\" (UID: \"8849f8ab-2b78-439c-911c-323c8c21c8b3\") " Sep 11 14:32:44.832394 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.832059 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "8849f8ab-2b78-439c-911c-323c8c21c8b3" (UID: "8849f8ab-2b78-439c-911c-323c8c21c8b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:44.832394 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.832149 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8849f8ab-2b78-439c-911c-323c8c21c8b3" (UID: "8849f8ab-2b78-439c-911c-323c8c21c8b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:44.832394 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.832285 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8849f8ab-2b78-439c-911c-323c8c21c8b3" (UID: "8849f8ab-2b78-439c-911c-323c8c21c8b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:44.832394 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.832301 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8849f8ab-2b78-439c-911c-323c8c21c8b3" (UID: "8849f8ab-2b78-439c-911c-323c8c21c8b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:44.832546 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.832396 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8849f8ab-2b78-439c-911c-323c8c21c8b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8849f8ab-2b78-439c-911c-323c8c21c8b3" (UID: "8849f8ab-2b78-439c-911c-323c8c21c8b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:32:44.832546 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.832424 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:44.832546 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.832442 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:44.832546 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.832452 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:44.832546 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.832461 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:44.834359 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.834327 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "8849f8ab-2b78-439c-911c-323c8c21c8b3" (UID: "8849f8ab-2b78-439c-911c-323c8c21c8b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:44.834490 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.834430 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8849f8ab-2b78-439c-911c-323c8c21c8b3" (UID: "8849f8ab-2b78-439c-911c-323c8c21c8b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:32:44.834749 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.834730 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8849f8ab-2b78-439c-911c-323c8c21c8b3-kube-api-access-8qk4s" (OuterVolumeSpecName: "kube-api-access-8qk4s") pod "8849f8ab-2b78-439c-911c-323c8c21c8b3" (UID: "8849f8ab-2b78-439c-911c-323c8c21c8b3"). InnerVolumeSpecName "kube-api-access-8qk4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:44.834749 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.834737 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8849f8ab-2b78-439c-911c-323c8c21c8b3" (UID: "8849f8ab-2b78-439c-911c-323c8c21c8b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:32:44.933847 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.933809 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:44.933847 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.933840 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:44.933847 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.933852 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8849f8ab-2b78-439c-911c-323c8c21c8b3-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:44.934132 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.933863 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qk4s\" (UniqueName: \"kubernetes.io/projected/8849f8ab-2b78-439c-911c-323c8c21c8b3-kube-api-access-8qk4s\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:44.934132 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:44.933876 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8849f8ab-2b78-439c-911c-323c8c21c8b3-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:32:45.120919 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:45.120819 2566 generic.go:356] "Generic (PLEG): container finished" podID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerID="30c1f82f596d356199794ba8bd99e298dc2ad4ba0f2fe2a9db5fde11dac75044" exitCode=0 Sep 11 14:32:45.120919 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:45.120879 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" event={"ID":"8849f8ab-2b78-439c-911c-323c8c21c8b3","Type":"ContainerDied","Data":"30c1f82f596d356199794ba8bd99e298dc2ad4ba0f2fe2a9db5fde11dac75044"} Sep 11 14:32:45.121160 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:45.120938 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" event={"ID":"8849f8ab-2b78-439c-911c-323c8c21c8b3","Type":"ContainerDied","Data":"4447723e9abf853b7739080134c71e4a7841d2acf91996af9ed744c5bc23138b"} Sep 11 14:32:45.121160 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:45.120960 2566 scope.go:117] "RemoveContainer" containerID="30c1f82f596d356199794ba8bd99e298dc2ad4ba0f2fe2a9db5fde11dac75044" Sep 11 14:32:45.121160 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:45.120889 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw" Sep 11 14:32:45.131589 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:45.131570 2566 scope.go:117] "RemoveContainer" containerID="4d7047beb7446609439de1e9b469e39aecd1a293a92854a7c725a86bfcf138dd" Sep 11 14:32:45.141339 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:45.141317 2566 scope.go:117] "RemoveContainer" containerID="30c1f82f596d356199794ba8bd99e298dc2ad4ba0f2fe2a9db5fde11dac75044" Sep 11 14:32:45.141614 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:32:45.141593 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c1f82f596d356199794ba8bd99e298dc2ad4ba0f2fe2a9db5fde11dac75044\": container with ID starting with 30c1f82f596d356199794ba8bd99e298dc2ad4ba0f2fe2a9db5fde11dac75044 not found: ID does not exist" containerID="30c1f82f596d356199794ba8bd99e298dc2ad4ba0f2fe2a9db5fde11dac75044" Sep 11 14:32:45.141699 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:45.141628 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c1f82f596d356199794ba8bd99e298dc2ad4ba0f2fe2a9db5fde11dac75044"} err="failed to get container status \"30c1f82f596d356199794ba8bd99e298dc2ad4ba0f2fe2a9db5fde11dac75044\": rpc error: code = NotFound desc = could not find container \"30c1f82f596d356199794ba8bd99e298dc2ad4ba0f2fe2a9db5fde11dac75044\": container with ID starting with 30c1f82f596d356199794ba8bd99e298dc2ad4ba0f2fe2a9db5fde11dac75044 not found: ID does not exist" Sep 11 14:32:45.141699 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:45.141653 2566 scope.go:117] "RemoveContainer" containerID="4d7047beb7446609439de1e9b469e39aecd1a293a92854a7c725a86bfcf138dd" Sep 11 14:32:45.141941 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:32:45.141920 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7047beb7446609439de1e9b469e39aecd1a293a92854a7c725a86bfcf138dd\": container with ID starting with 4d7047beb7446609439de1e9b469e39aecd1a293a92854a7c725a86bfcf138dd not found: ID does not exist" containerID="4d7047beb7446609439de1e9b469e39aecd1a293a92854a7c725a86bfcf138dd" Sep 11 14:32:45.141983 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:45.141948 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7047beb7446609439de1e9b469e39aecd1a293a92854a7c725a86bfcf138dd"} err="failed to get container status \"4d7047beb7446609439de1e9b469e39aecd1a293a92854a7c725a86bfcf138dd\": rpc error: code = NotFound desc = could not find container \"4d7047beb7446609439de1e9b469e39aecd1a293a92854a7c725a86bfcf138dd\": container with ID starting with 4d7047beb7446609439de1e9b469e39aecd1a293a92854a7c725a86bfcf138dd not found: ID does not exist" Sep 11 14:32:45.146569 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:45.146548 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw"] Sep 11 14:32:45.152195 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:45.152172 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1e2e7d-openshift-default-7dd798d7c6-bp5vw"] Sep 11 14:32:46.098612 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:46.098575 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" path="/var/lib/kubelet/pods/8849f8ab-2b78-439c-911c-323c8c21c8b3/volumes" Sep 11 14:32:50.668157 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:50.668125 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-031a56-29818950-k5rnw"] Sep 11 14:32:50.674671 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:50.674643 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-031a56-29818950-k5rnw"] Sep 11 14:32:52.098646 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:52.098612 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d6f820e-97ab-4b22-8a85-560283f275cc" path="/var/lib/kubelet/pods/6d6f820e-97ab-4b22-8a85-560283f275cc/volumes" Sep 11 14:32:58.106229 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:58.106196 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5"] Sep 11 14:32:58.106676 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:32:58.106393 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" podUID="adb57f87-d90c-421c-9751-70026534903d" containerName="istio-proxy" containerID="cri-o://8d7ef5a349b20b9da7f75c7a841c0b545925d4708f5117cda8789f47661a5af7" gracePeriod=30 Sep 11 14:33:03.241241 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.241213 2566 generic.go:356] "Generic (PLEG): container finished" podID="adb57f87-d90c-421c-9751-70026534903d" containerID="8d7ef5a349b20b9da7f75c7a841c0b545925d4708f5117cda8789f47661a5af7" exitCode=0 Sep 11 14:33:03.241694 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.241294 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" event={"ID":"adb57f87-d90c-421c-9751-70026534903d","Type":"ContainerDied","Data":"8d7ef5a349b20b9da7f75c7a841c0b545925d4708f5117cda8789f47661a5af7"} Sep 11 14:33:03.241925 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.241903 2566 scope.go:117] "RemoveContainer" containerID="cf4c0aeee0043b2eea346a3667cd18246b4f4f11cc5309d9c3900fa4cac91eb8" Sep 11 14:33:03.386027 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.385966 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:33:03.404509 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.404472 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb57f87-d90c-421c-9751-70026534903d-istio-token\") pod \"adb57f87-d90c-421c-9751-70026534903d\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " Sep 11 14:33:03.404509 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.404503 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-workload-socket\") pod \"adb57f87-d90c-421c-9751-70026534903d\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " Sep 11 14:33:03.404730 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.404535 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-credential-socket\") pod \"adb57f87-d90c-421c-9751-70026534903d\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " Sep 11 14:33:03.404730 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.404593 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-workload-certs\") pod \"adb57f87-d90c-421c-9751-70026534903d\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " Sep 11 14:33:03.404730 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.404619 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-istio-data\") pod \"adb57f87-d90c-421c-9751-70026534903d\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " Sep 11 14:33:03.404730 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.404654 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb57f87-d90c-421c-9751-70026534903d-istiod-ca-cert\") pod \"adb57f87-d90c-421c-9751-70026534903d\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " Sep 11 14:33:03.404730 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.404680 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq5fp\" (UniqueName: \"kubernetes.io/projected/adb57f87-d90c-421c-9751-70026534903d-kube-api-access-gq5fp\") pod \"adb57f87-d90c-421c-9751-70026534903d\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " Sep 11 14:33:03.404730 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.404720 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-istio-envoy\") pod \"adb57f87-d90c-421c-9751-70026534903d\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " Sep 11 14:33:03.405178 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.404764 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb57f87-d90c-421c-9751-70026534903d-istio-podinfo\") pod \"adb57f87-d90c-421c-9751-70026534903d\" (UID: \"adb57f87-d90c-421c-9751-70026534903d\") " Sep 11 14:33:03.405594 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.405559 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-istio-data" (OuterVolumeSpecName: "istio-data") pod "adb57f87-d90c-421c-9751-70026534903d" (UID: "adb57f87-d90c-421c-9751-70026534903d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:33:03.406707 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.406014 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "adb57f87-d90c-421c-9751-70026534903d" (UID: "adb57f87-d90c-421c-9751-70026534903d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:33:03.406707 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.406110 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "adb57f87-d90c-421c-9751-70026534903d" (UID: "adb57f87-d90c-421c-9751-70026534903d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:33:03.406707 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.406664 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb57f87-d90c-421c-9751-70026534903d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "adb57f87-d90c-421c-9751-70026534903d" (UID: "adb57f87-d90c-421c-9751-70026534903d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:33:03.407398 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.407378 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "adb57f87-d90c-421c-9751-70026534903d" (UID: "adb57f87-d90c-421c-9751-70026534903d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:33:03.408172 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.407816 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adb57f87-d90c-421c-9751-70026534903d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "adb57f87-d90c-421c-9751-70026534903d" (UID: "adb57f87-d90c-421c-9751-70026534903d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:33:03.408172 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.408132 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb57f87-d90c-421c-9751-70026534903d-istio-token" (OuterVolumeSpecName: "istio-token") pod "adb57f87-d90c-421c-9751-70026534903d" (UID: "adb57f87-d90c-421c-9751-70026534903d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:03.409166 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.409140 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "adb57f87-d90c-421c-9751-70026534903d" (UID: "adb57f87-d90c-421c-9751-70026534903d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:33:03.409402 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.409373 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb57f87-d90c-421c-9751-70026534903d-kube-api-access-gq5fp" (OuterVolumeSpecName: "kube-api-access-gq5fp") pod "adb57f87-d90c-421c-9751-70026534903d" (UID: "adb57f87-d90c-421c-9751-70026534903d"). InnerVolumeSpecName "kube-api-access-gq5fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:03.506626 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.506583 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb57f87-d90c-421c-9751-70026534903d-istiod-ca-cert\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:33:03.506626 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.506625 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq5fp\" (UniqueName: \"kubernetes.io/projected/adb57f87-d90c-421c-9751-70026534903d-kube-api-access-gq5fp\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:33:03.506927 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.506641 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-istio-envoy\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:33:03.506927 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.506658 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb57f87-d90c-421c-9751-70026534903d-istio-podinfo\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:33:03.506927 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.506674 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb57f87-d90c-421c-9751-70026534903d-istio-token\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:33:03.506927 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.506687 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-workload-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:33:03.506927 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.506700 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-credential-socket\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:33:03.506927 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.506714 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-workload-certs\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:33:03.506927 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:03.506729 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb57f87-d90c-421c-9751-70026534903d-istio-data\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:33:04.249104 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:04.249064 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" Sep 11 14:33:04.249544 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:04.249057 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5" event={"ID":"adb57f87-d90c-421c-9751-70026534903d","Type":"ContainerDied","Data":"6a3fe3b455da864101d939b036dfb55e221f4db804bdda747d67d6555cdac5a8"} Sep 11 14:33:04.249544 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:04.249234 2566 scope.go:117] "RemoveContainer" containerID="8d7ef5a349b20b9da7f75c7a841c0b545925d4708f5117cda8789f47661a5af7" Sep 11 14:33:04.279351 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:04.279311 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5"] Sep 11 14:33:04.288671 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:04.288641 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-031a56-openshift-default-5487d645cd-9vgs5"] Sep 11 14:33:06.097776 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:33:06.097742 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adb57f87-d90c-421c-9751-70026534903d" path="/var/lib/kubelet/pods/adb57f87-d90c-421c-9751-70026534903d/volumes" Sep 11 14:35:31.695971 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:31.695932 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd"] Sep 11 14:35:31.698454 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:31.696162 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" podUID="264ffa70-1a57-4574-8e9c-7803d5260a37" containerName="manager" containerID="cri-o://dc9959f0d0f00d058dcf2aa2bedcd27e1ce1c131440cb0c9ad4c5ba4696c0d90" gracePeriod=10 Sep 11 14:35:34.055463 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.055438 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" Sep 11 14:35:34.153590 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.153561 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b6qk\" (UniqueName: \"kubernetes.io/projected/264ffa70-1a57-4574-8e9c-7803d5260a37-kube-api-access-8b6qk\") pod \"264ffa70-1a57-4574-8e9c-7803d5260a37\" (UID: \"264ffa70-1a57-4574-8e9c-7803d5260a37\") " Sep 11 14:35:34.153759 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.153598 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/264ffa70-1a57-4574-8e9c-7803d5260a37-extensions-socket-volume\") pod \"264ffa70-1a57-4574-8e9c-7803d5260a37\" (UID: \"264ffa70-1a57-4574-8e9c-7803d5260a37\") " Sep 11 14:35:34.154018 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.153995 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264ffa70-1a57-4574-8e9c-7803d5260a37-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "264ffa70-1a57-4574-8e9c-7803d5260a37" (UID: "264ffa70-1a57-4574-8e9c-7803d5260a37"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:35:34.155781 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.155758 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264ffa70-1a57-4574-8e9c-7803d5260a37-kube-api-access-8b6qk" (OuterVolumeSpecName: "kube-api-access-8b6qk") pod "264ffa70-1a57-4574-8e9c-7803d5260a37" (UID: "264ffa70-1a57-4574-8e9c-7803d5260a37"). InnerVolumeSpecName "kube-api-access-8b6qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:34.165500 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.165476 2566 generic.go:356] "Generic (PLEG): container finished" podID="264ffa70-1a57-4574-8e9c-7803d5260a37" containerID="dc9959f0d0f00d058dcf2aa2bedcd27e1ce1c131440cb0c9ad4c5ba4696c0d90" exitCode=0 Sep 11 14:35:34.165633 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.165525 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" Sep 11 14:35:34.165633 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.165565 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" event={"ID":"264ffa70-1a57-4574-8e9c-7803d5260a37","Type":"ContainerDied","Data":"dc9959f0d0f00d058dcf2aa2bedcd27e1ce1c131440cb0c9ad4c5ba4696c0d90"} Sep 11 14:35:34.165633 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.165620 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd" event={"ID":"264ffa70-1a57-4574-8e9c-7803d5260a37","Type":"ContainerDied","Data":"735dbd783f96e9baab2a65da449f4289998b095b771e377776d3901acb9fa41e"} Sep 11 14:35:34.165787 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.165642 2566 scope.go:117] "RemoveContainer" containerID="dc9959f0d0f00d058dcf2aa2bedcd27e1ce1c131440cb0c9ad4c5ba4696c0d90" Sep 11 14:35:34.175545 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.175529 2566 scope.go:117] "RemoveContainer" containerID="dc9959f0d0f00d058dcf2aa2bedcd27e1ce1c131440cb0c9ad4c5ba4696c0d90" Sep 11 14:35:34.175826 ip-10-0-136-92 kubenswrapper[2566]: E0911 14:35:34.175807 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9959f0d0f00d058dcf2aa2bedcd27e1ce1c131440cb0c9ad4c5ba4696c0d90\": container with ID starting with dc9959f0d0f00d058dcf2aa2bedcd27e1ce1c131440cb0c9ad4c5ba4696c0d90 not found: ID does not exist" containerID="dc9959f0d0f00d058dcf2aa2bedcd27e1ce1c131440cb0c9ad4c5ba4696c0d90" Sep 11 14:35:34.175881 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.175834 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9959f0d0f00d058dcf2aa2bedcd27e1ce1c131440cb0c9ad4c5ba4696c0d90"} err="failed to get container status \"dc9959f0d0f00d058dcf2aa2bedcd27e1ce1c131440cb0c9ad4c5ba4696c0d90\": rpc error: code = NotFound desc = could not find container \"dc9959f0d0f00d058dcf2aa2bedcd27e1ce1c131440cb0c9ad4c5ba4696c0d90\": container with ID starting with dc9959f0d0f00d058dcf2aa2bedcd27e1ce1c131440cb0c9ad4c5ba4696c0d90 not found: ID does not exist" Sep 11 14:35:34.189176 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.189154 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd"] Sep 11 14:35:34.194968 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.194949 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v9csd"] Sep 11 14:35:34.254557 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.254515 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b6qk\" (UniqueName: \"kubernetes.io/projected/264ffa70-1a57-4574-8e9c-7803d5260a37-kube-api-access-8b6qk\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:35:34.254557 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:34.254548 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/264ffa70-1a57-4574-8e9c-7803d5260a37-extensions-socket-volume\") on node \"ip-10-0-136-92.ec2.internal\" DevicePath \"\"" Sep 11 14:35:36.098616 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:35:36.098572 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="264ffa70-1a57-4574-8e9c-7803d5260a37" path="/var/lib/kubelet/pods/264ffa70-1a57-4574-8e9c-7803d5260a37/volumes" Sep 11 14:36:32.416897 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.416814 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx"] Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417217 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb57f87-d90c-421c-9751-70026534903d" containerName="istio-proxy" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417230 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb57f87-d90c-421c-9751-70026534903d" containerName="istio-proxy" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417238 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb57f87-d90c-421c-9751-70026534903d" containerName="istio-proxy" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417243 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb57f87-d90c-421c-9751-70026534903d" containerName="istio-proxy" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417251 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17db22a-8bd6-4571-a575-1b160cf26ff2" containerName="istio-proxy" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417257 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17db22a-8bd6-4571-a575-1b160cf26ff2" containerName="istio-proxy" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417264 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" containerName="istio-proxy" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417269 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdfd40a0-7939-4f88-aca1-d10e2d6d10c0" containerName="istio-proxy" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417277 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerName="istio-proxy" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417283 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerName="istio-proxy" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417291 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" containerName="istio-proxy" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417296 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" containerName="istio-proxy" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417303 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17db22a-8bd6-4571-a575-1b160cf26ff2" containerName="istio-proxy" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417308 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17db22a-8bd6-4571-a575-1b160cf26ff2" containerName="istio-proxy" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417319 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63ff19ef-39d2-44f3-90f4-178a59eee550" containerName="revoke-keys" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417323 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="63ff19ef-39d2-44f3-90f4-178a59eee550" containerName="revoke-keys" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417333 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264ffa70-1a57-4574-8e9c-7803d5260a37" containerName="manager" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417337 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="264ffa70-1a57-4574-8e9c-7803d5260a37" containerName="manager" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417347 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" containerName="istio-proxy" Sep 11 14:36:32.417377 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417352 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" containerName="istio-proxy" Sep 11 14:36:32.417981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417412 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerName="istio-proxy" Sep 11 14:36:32.417981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417420 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerName="istio-proxy" Sep 11 14:36:32.417981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417427 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a17db22a-8bd6-4571-a575-1b160cf26ff2" containerName="istio-proxy" Sep 11 14:36:32.417981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417434 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb57f87-d90c-421c-9751-70026534903d" containerName="istio-proxy" Sep 11 14:36:32.417981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417440 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" containerName="istio-proxy" Sep 11 14:36:32.417981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417445 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="264ffa70-1a57-4574-8e9c-7803d5260a37" containerName="manager" Sep 11 14:36:32.417981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417451 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb57f87-d90c-421c-9751-70026534903d" containerName="istio-proxy" Sep 11 14:36:32.417981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417456 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a17db22a-8bd6-4571-a575-1b160cf26ff2" containerName="istio-proxy" Sep 11 14:36:32.417981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417461 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="63ff19ef-39d2-44f3-90f4-178a59eee550" containerName="revoke-keys" Sep 11 14:36:32.417981 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.417466 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa5f5af-9d1f-4095-96e5-0e64f52fde1f" containerName="istio-proxy" Sep 11 14:36:32.420361 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.420345 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx" Sep 11 14:36:32.424251 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.424229 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l9xxt\"" Sep 11 14:36:32.440797 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.440766 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx"] Sep 11 14:36:32.482358 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.482328 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbwjp\" (UniqueName: \"kubernetes.io/projected/5a69fd34-8856-45d5-b2a6-108e1c7f3159-kube-api-access-sbwjp\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5ccgx\" (UID: \"5a69fd34-8856-45d5-b2a6-108e1c7f3159\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx" Sep 11 14:36:32.482547 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.482386 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a69fd34-8856-45d5-b2a6-108e1c7f3159-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5ccgx\" (UID: \"5a69fd34-8856-45d5-b2a6-108e1c7f3159\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx" Sep 11 14:36:32.583011 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.582967 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbwjp\" (UniqueName: \"kubernetes.io/projected/5a69fd34-8856-45d5-b2a6-108e1c7f3159-kube-api-access-sbwjp\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5ccgx\" (UID: \"5a69fd34-8856-45d5-b2a6-108e1c7f3159\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx" Sep 11 14:36:32.583239 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.583024 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a69fd34-8856-45d5-b2a6-108e1c7f3159-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5ccgx\" (UID: \"5a69fd34-8856-45d5-b2a6-108e1c7f3159\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx" Sep 11 14:36:32.583476 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.583451 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a69fd34-8856-45d5-b2a6-108e1c7f3159-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5ccgx\" (UID: \"5a69fd34-8856-45d5-b2a6-108e1c7f3159\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx" Sep 11 14:36:32.604772 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.604744 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbwjp\" (UniqueName: \"kubernetes.io/projected/5a69fd34-8856-45d5-b2a6-108e1c7f3159-kube-api-access-sbwjp\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5ccgx\" (UID: \"5a69fd34-8856-45d5-b2a6-108e1c7f3159\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx" Sep 11 14:36:32.730068 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:32.730036 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx" Sep 11 14:36:33.014915 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:33.014889 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx"] Sep 11 14:36:33.016971 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:36:33.016939 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a69fd34_8856_45d5_b2a6_108e1c7f3159.slice/crio-cbc4f47a63ce785353aafbc908264f78916411fdbb105c0b0b0b6a188c160c37 WatchSource:0}: Error finding container cbc4f47a63ce785353aafbc908264f78916411fdbb105c0b0b0b6a188c160c37: Status 404 returned error can't find the container with id cbc4f47a63ce785353aafbc908264f78916411fdbb105c0b0b0b6a188c160c37 Sep 11 14:36:33.526109 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:33.526053 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx" event={"ID":"5a69fd34-8856-45d5-b2a6-108e1c7f3159","Type":"ContainerStarted","Data":"6ad732bf3907bbfa1e06026ead221ae2726fd677a749ff226c1ebd927a79041c"} Sep 11 14:36:33.526509 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:33.526125 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx" event={"ID":"5a69fd34-8856-45d5-b2a6-108e1c7f3159","Type":"ContainerStarted","Data":"cbc4f47a63ce785353aafbc908264f78916411fdbb105c0b0b0b6a188c160c37"} Sep 11 14:36:33.526509 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:33.526281 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx" Sep 11 14:36:33.594621 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:33.594561 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx" podStartSLOduration=1.594539026 podStartE2EDuration="1.594539026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:36:33.591736413 +0000 UTC m=+2333.996717542" watchObservedRunningTime="2026-09-11 14:36:33.594539026 +0000 UTC m=+2333.999520153" Sep 11 14:36:40.761486 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:40.761452 2566 scope.go:117] "RemoveContainer" containerID="3358acf7ebd0faaa7ea26b089c42cd8070f04bff3e5c6d875a18d6c4f51144ef" Sep 11 14:36:40.770814 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:40.770785 2566 scope.go:117] "RemoveContainer" containerID="313e6e2ee9c9c95325126b35bbfa7ed5fcc47cb4bb2dda8f967306d4cde15b25" Sep 11 14:36:40.779107 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:40.779066 2566 scope.go:117] "RemoveContainer" containerID="11f718dc8d448a341e259a8463d562c1cb581bbf8354a02c5bc80b2ff723b970" Sep 11 14:36:40.786957 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:40.786938 2566 scope.go:117] "RemoveContainer" containerID="945597ac1ef3d9047cc8b3258618dda8cc13a10a2290ff1884924d295d786997" Sep 11 14:36:44.535554 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:36:44.535524 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5ccgx" Sep 11 14:37:40.324867 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:37:40.324828 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:37:40.332215 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:37:40.332192 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:37:40.820951 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:37:40.820916 2566 scope.go:117] "RemoveContainer" containerID="a2580b4897b898514923637c76d6e1622c45f6da228c3c148165c3dfb77d1ba8" Sep 11 14:38:26.125754 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:26.125669 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bd69597d-7sb7p_c8a29ab4-b886-48d2-9579-3e3a1dba2a78/authorino/0.log" Sep 11 14:38:31.656834 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:31.656796 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jxzrs_2b4bed51-642c-46d6-a9f7-a1184b6d21be/manager/0.log" Sep 11 14:38:31.784552 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:31.784516 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vwzkb_922f7c36-5337-45e2-a7b6-94f1ced90673/manager/0.log" Sep 11 14:38:32.549222 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:32.549195 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586bcdcf6-z2w7n_b8119fc8-fece-463f-8abb-c549cbbb0126/copy-manifests/0.log" Sep 11 14:38:32.568267 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:32.568230 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586bcdcf6-z2w7n_b8119fc8-fece-463f-8abb-c549cbbb0126/manager/0.log" Sep 11 14:38:33.150330 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:33.150297 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818935-c99c9_4c8551ad-387a-468c-9b5e-eb10a7df4b21/cleanup/0.log" Sep 11 14:38:33.385427 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:33.385382 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-s8dp6_90862da7-61de-4138-a7fb-7c2787d43ae8/postgres/0.log" Sep 11 14:38:34.208595 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:34.208564 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bd69597d-7sb7p_c8a29ab4-b886-48d2-9579-3e3a1dba2a78/authorino/0.log" Sep 11 14:38:34.339588 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:34.339547 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-cw6lr_44ea994f-1c5d-4fa0-8029-14c347f2df55/manager/0.log" Sep 11 14:38:34.721103 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:34.721038 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-5ccgx_5a69fd34-8856-45d5-b2a6-108e1c7f3159/manager/0.log" Sep 11 14:38:34.991796 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:34.991757 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-vpfjn_ffcecabb-7b78-4fb8-99ed-3918f9b1db5b/manager/0.log" Sep 11 14:38:35.382358 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:35.382261 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht_4f08071d-1a99-4cd9-8929-f202cd23051d/istio-proxy/0.log" Sep 11 14:38:35.741230 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:35.741192 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8494df7896-tszz7_45ab0a34-6a08-4cc3-9754-549294bd0b74/kube-auth-proxy/0.log" Sep 11 14:38:35.872232 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:35.872200 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-t98fx_31429fc7-4a21-4e5d-9db7-0a0663bf8a71/istio-proxy/1.log" Sep 11 14:38:36.133730 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:36.133644 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-rjfcb_417e7a61-fc2b-4125-97cf-53ef51e93982/payload-processing/0.log" Sep 11 14:38:36.726657 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:36.726630 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-x5m7n_4b2883f5-5b16-49be-8540-cb7d7967af6a/main/0.log" Sep 11 14:38:36.980588 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:36.980516 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-sscpl_62bcf888-fbca-4262-9906-2bc49aa00b85/main/0.log" Sep 11 14:38:40.844798 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:40.844761 2566 scope.go:117] "RemoveContainer" containerID="9f9b6127e17a3d82833936ffc74f69396ea22f9c607964088b8813e7db8261d5" Sep 11 14:38:42.577462 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:42.577431 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ktch4/must-gather-v9fr4"] Sep 11 14:38:42.577847 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:42.577819 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerName="istio-proxy" Sep 11 14:38:42.577847 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:42.577830 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8849f8ab-2b78-439c-911c-323c8c21c8b3" containerName="istio-proxy" Sep 11 14:38:42.582719 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:42.582687 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ktch4/must-gather-v9fr4" Sep 11 14:38:42.583596 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:42.583570 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/522ae7dc-d875-4d48-a2bd-3c2c818f0b7b-must-gather-output\") pod \"must-gather-v9fr4\" (UID: \"522ae7dc-d875-4d48-a2bd-3c2c818f0b7b\") " pod="openshift-must-gather-ktch4/must-gather-v9fr4" Sep 11 14:38:42.583682 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:42.583628 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tb6p\" (UniqueName: \"kubernetes.io/projected/522ae7dc-d875-4d48-a2bd-3c2c818f0b7b-kube-api-access-2tb6p\") pod \"must-gather-v9fr4\" (UID: \"522ae7dc-d875-4d48-a2bd-3c2c818f0b7b\") " pod="openshift-must-gather-ktch4/must-gather-v9fr4" Sep 11 14:38:42.586928 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:42.586912 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ktch4\"/\"kube-root-ca.crt\"" Sep 11 14:38:42.587067 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:42.587040 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ktch4\"/\"openshift-service-ca.crt\"" Sep 11 14:38:42.594109 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:42.594068 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ktch4/must-gather-v9fr4"] Sep 11 14:38:42.684694 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:42.684656 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/522ae7dc-d875-4d48-a2bd-3c2c818f0b7b-must-gather-output\") pod \"must-gather-v9fr4\" (UID: \"522ae7dc-d875-4d48-a2bd-3c2c818f0b7b\") " pod="openshift-must-gather-ktch4/must-gather-v9fr4" Sep 11 14:38:42.684880 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:42.684714 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tb6p\" (UniqueName: \"kubernetes.io/projected/522ae7dc-d875-4d48-a2bd-3c2c818f0b7b-kube-api-access-2tb6p\") pod \"must-gather-v9fr4\" (UID: \"522ae7dc-d875-4d48-a2bd-3c2c818f0b7b\") " pod="openshift-must-gather-ktch4/must-gather-v9fr4" Sep 11 14:38:42.685032 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:42.685014 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/522ae7dc-d875-4d48-a2bd-3c2c818f0b7b-must-gather-output\") pod \"must-gather-v9fr4\" (UID: \"522ae7dc-d875-4d48-a2bd-3c2c818f0b7b\") " pod="openshift-must-gather-ktch4/must-gather-v9fr4" Sep 11 14:38:42.693708 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:42.693678 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tb6p\" (UniqueName: \"kubernetes.io/projected/522ae7dc-d875-4d48-a2bd-3c2c818f0b7b-kube-api-access-2tb6p\") pod \"must-gather-v9fr4\" (UID: \"522ae7dc-d875-4d48-a2bd-3c2c818f0b7b\") " pod="openshift-must-gather-ktch4/must-gather-v9fr4" Sep 11 14:38:42.892665 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:42.892582 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ktch4/must-gather-v9fr4" Sep 11 14:38:43.177507 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:43.177479 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ktch4/must-gather-v9fr4"] Sep 11 14:38:43.179616 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:38:43.179592 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod522ae7dc_d875_4d48_a2bd_3c2c818f0b7b.slice/crio-33f6a48cc3ddacaa316abce7fa61c6484aff14a143126aa221f057a194e10c37 WatchSource:0}: Error finding container 33f6a48cc3ddacaa316abce7fa61c6484aff14a143126aa221f057a194e10c37: Status 404 returned error can't find the container with id 33f6a48cc3ddacaa316abce7fa61c6484aff14a143126aa221f057a194e10c37 Sep 11 14:38:43.181405 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:43.181388 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:38:43.304055 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:43.304013 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ktch4/must-gather-v9fr4" event={"ID":"522ae7dc-d875-4d48-a2bd-3c2c818f0b7b","Type":"ContainerStarted","Data":"33f6a48cc3ddacaa316abce7fa61c6484aff14a143126aa221f057a194e10c37"} Sep 11 14:38:44.321260 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:44.320769 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ktch4/must-gather-v9fr4" event={"ID":"522ae7dc-d875-4d48-a2bd-3c2c818f0b7b","Type":"ContainerStarted","Data":"f230f818972a7a24ae8cf18007057010b0bb91fbdaaac7123d3939164bb74e8c"} Sep 11 14:38:44.321260 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:44.320831 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ktch4/must-gather-v9fr4" event={"ID":"522ae7dc-d875-4d48-a2bd-3c2c818f0b7b","Type":"ContainerStarted","Data":"c7a14154ece325fbbb3ac000ccc21ef1db23e35c600a8e1538b0fe8855c70471"} Sep 11 14:38:44.341856 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:44.341804 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ktch4/must-gather-v9fr4" podStartSLOduration=1.559735085 podStartE2EDuration="2.341787543s" totalImagesPullingTime="782.052458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:44.338295218 +0000 UTC m=+2464.743276345" watchObservedRunningTime="2026-09-11 14:38:44.341787543 +0000 UTC m=+2464.746768671" Sep 11 14:38:45.572502 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:45.572468 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4wl44_3b82ace2-23c3-42cc-ae13-6b30c7ebe5b9/global-pull-secret-syncer/0.log" Sep 11 14:38:45.819984 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:45.819936 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zvnz6_9ec98f86-dcb9-42cf-9add-e2045fbbe773/konnectivity-agent/0.log" Sep 11 14:38:45.876056 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:45.875979 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-92.ec2.internal_504a2a5e8c9cf487f28426e21025e719/haproxy/0.log" Sep 11 14:38:50.657444 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:50.657401 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5bd69597d-7sb7p_c8a29ab4-b886-48d2-9579-3e3a1dba2a78/authorino/0.log" Sep 11 14:38:50.705804 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:50.705765 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-cw6lr_44ea994f-1c5d-4fa0-8029-14c347f2df55/manager/0.log" Sep 11 14:38:50.825502 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:50.825462 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-5ccgx_5a69fd34-8856-45d5-b2a6-108e1c7f3159/manager/0.log" Sep 11 14:38:50.964161 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:50.962035 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-vpfjn_ffcecabb-7b78-4fb8-99ed-3918f9b1db5b/manager/0.log" Sep 11 14:38:52.928293 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:52.928244 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-989k8_e83c0e3b-9a72-4b5b-af23-5a5f61680944/monitoring-plugin/0.log" Sep 11 14:38:53.007611 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:53.007582 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8qmlv_09f40fed-0af8-4fc9-8b29-fcb98e52e8c6/node-exporter/0.log" Sep 11 14:38:53.073327 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:53.073297 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8qmlv_09f40fed-0af8-4fc9-8b29-fcb98e52e8c6/kube-rbac-proxy/0.log" Sep 11 14:38:53.113365 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:53.113334 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8qmlv_09f40fed-0af8-4fc9-8b29-fcb98e52e8c6/init-textfile/0.log" Sep 11 14:38:53.673109 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:53.673016 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-8cjb2_c0038abf-1e13-4181-967c-a008f211149f/prometheus-operator/0.log" Sep 11 14:38:53.701051 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:53.701019 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-8cjb2_c0038abf-1e13-4181-967c-a008f211149f/kube-rbac-proxy/0.log" Sep 11 14:38:53.741811 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:53.741785 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-mlgdz_f810a603-3b8d-498a-9b13-dcf15aed909e/prometheus-operator-admission-webhook/0.log" Sep 11 14:38:53.882246 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:53.882215 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ffffdd878-wn7tn_dd1efa0e-bbe9-41a1-9b22-16cc926202a5/thanos-query/0.log" Sep 11 14:38:53.912562 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:53.912518 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ffffdd878-wn7tn_dd1efa0e-bbe9-41a1-9b22-16cc926202a5/kube-rbac-proxy-web/0.log" Sep 11 14:38:53.955159 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:53.954549 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ffffdd878-wn7tn_dd1efa0e-bbe9-41a1-9b22-16cc926202a5/kube-rbac-proxy/0.log" Sep 11 14:38:53.979209 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:53.979111 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ffffdd878-wn7tn_dd1efa0e-bbe9-41a1-9b22-16cc926202a5/prom-label-proxy/0.log" Sep 11 14:38:54.007406 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:54.007321 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ffffdd878-wn7tn_dd1efa0e-bbe9-41a1-9b22-16cc926202a5/kube-rbac-proxy-rules/0.log" Sep 11 14:38:54.035381 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:54.035308 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ffffdd878-wn7tn_dd1efa0e-bbe9-41a1-9b22-16cc926202a5/kube-rbac-proxy-metrics/0.log" Sep 11 14:38:55.629957 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.629926 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh"] Sep 11 14:38:55.635646 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.635617 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.640460 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.640426 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ktch4\"/\"default-dockercfg-mxzzh\"" Sep 11 14:38:55.648455 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.648427 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh"] Sep 11 14:38:55.742110 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.741061 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cc756bf-7a14-43ea-a25e-3ef55183f3fa-lib-modules\") pod \"perf-node-gather-daemonset-27pqh\" (UID: \"3cc756bf-7a14-43ea-a25e-3ef55183f3fa\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.742110 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.741136 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3cc756bf-7a14-43ea-a25e-3ef55183f3fa-proc\") pod \"perf-node-gather-daemonset-27pqh\" (UID: \"3cc756bf-7a14-43ea-a25e-3ef55183f3fa\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.742110 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.741167 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb5t4\" (UniqueName: \"kubernetes.io/projected/3cc756bf-7a14-43ea-a25e-3ef55183f3fa-kube-api-access-kb5t4\") pod \"perf-node-gather-daemonset-27pqh\" (UID: \"3cc756bf-7a14-43ea-a25e-3ef55183f3fa\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.742110 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.741205 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cc756bf-7a14-43ea-a25e-3ef55183f3fa-sys\") pod \"perf-node-gather-daemonset-27pqh\" (UID: \"3cc756bf-7a14-43ea-a25e-3ef55183f3fa\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.742110 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.741233 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3cc756bf-7a14-43ea-a25e-3ef55183f3fa-podres\") pod \"perf-node-gather-daemonset-27pqh\" (UID: \"3cc756bf-7a14-43ea-a25e-3ef55183f3fa\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.841985 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.841944 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cc756bf-7a14-43ea-a25e-3ef55183f3fa-lib-modules\") pod \"perf-node-gather-daemonset-27pqh\" (UID: \"3cc756bf-7a14-43ea-a25e-3ef55183f3fa\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.842183 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.842003 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3cc756bf-7a14-43ea-a25e-3ef55183f3fa-proc\") pod \"perf-node-gather-daemonset-27pqh\" (UID: \"3cc756bf-7a14-43ea-a25e-3ef55183f3fa\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.842183 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.842030 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb5t4\" (UniqueName: \"kubernetes.io/projected/3cc756bf-7a14-43ea-a25e-3ef55183f3fa-kube-api-access-kb5t4\") pod \"perf-node-gather-daemonset-27pqh\" (UID: \"3cc756bf-7a14-43ea-a25e-3ef55183f3fa\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.842183 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.842061 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cc756bf-7a14-43ea-a25e-3ef55183f3fa-sys\") pod \"perf-node-gather-daemonset-27pqh\" (UID: \"3cc756bf-7a14-43ea-a25e-3ef55183f3fa\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.842183 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.842117 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3cc756bf-7a14-43ea-a25e-3ef55183f3fa-podres\") pod \"perf-node-gather-daemonset-27pqh\" (UID: \"3cc756bf-7a14-43ea-a25e-3ef55183f3fa\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.842183 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.842155 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3cc756bf-7a14-43ea-a25e-3ef55183f3fa-lib-modules\") pod \"perf-node-gather-daemonset-27pqh\" (UID: \"3cc756bf-7a14-43ea-a25e-3ef55183f3fa\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.842183 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.842178 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3cc756bf-7a14-43ea-a25e-3ef55183f3fa-proc\") pod \"perf-node-gather-daemonset-27pqh\" (UID: \"3cc756bf-7a14-43ea-a25e-3ef55183f3fa\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.842481 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.842243 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3cc756bf-7a14-43ea-a25e-3ef55183f3fa-sys\") pod \"perf-node-gather-daemonset-27pqh\" (UID: \"3cc756bf-7a14-43ea-a25e-3ef55183f3fa\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.842481 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.842243 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3cc756bf-7a14-43ea-a25e-3ef55183f3fa-podres\") pod \"perf-node-gather-daemonset-27pqh\" (UID: \"3cc756bf-7a14-43ea-a25e-3ef55183f3fa\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.852328 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.852296 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb5t4\" (UniqueName: \"kubernetes.io/projected/3cc756bf-7a14-43ea-a25e-3ef55183f3fa-kube-api-access-kb5t4\") pod \"perf-node-gather-daemonset-27pqh\" (UID: \"3cc756bf-7a14-43ea-a25e-3ef55183f3fa\") " pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.955760 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.955670 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:55.976298 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.976114 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/2.log" Sep 11 14:38:55.981624 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:55.981593 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-qvnzs_6b7aadeb-5bf7-48b0-9169-7342cc8469ce/console-operator/3.log" Sep 11 14:38:56.257941 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:56.257912 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh"] Sep 11 14:38:56.260057 ip-10-0-136-92 kubenswrapper[2566]: W0911 14:38:56.260026 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3cc756bf_7a14_43ea_a25e_3ef55183f3fa.slice/crio-0dd4aa8a711f696250f656e0953222aaa4915a90fece85f9cfe600ccd61bc419 WatchSource:0}: Error finding container 0dd4aa8a711f696250f656e0953222aaa4915a90fece85f9cfe600ccd61bc419: Status 404 returned error can't find the container with id 0dd4aa8a711f696250f656e0953222aaa4915a90fece85f9cfe600ccd61bc419 Sep 11 14:38:56.426936 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:56.426894 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" event={"ID":"3cc756bf-7a14-43ea-a25e-3ef55183f3fa","Type":"ContainerStarted","Data":"66b6db0ae4862f8f4a7a1c3fa092c76031793a8ddc4bbf7e31aa6b212a3697b7"} Sep 11 14:38:56.427142 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:56.426964 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" event={"ID":"3cc756bf-7a14-43ea-a25e-3ef55183f3fa","Type":"ContainerStarted","Data":"0dd4aa8a711f696250f656e0953222aaa4915a90fece85f9cfe600ccd61bc419"} Sep 11 14:38:56.427826 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:56.427797 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:38:56.452781 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:56.452713 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" podStartSLOduration=1.452691013 podStartE2EDuration="1.452691013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:56.448814259 +0000 UTC m=+2476.853795387" watchObservedRunningTime="2026-09-11 14:38:56.452691013 +0000 UTC m=+2476.857672142" Sep 11 14:38:56.601938 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:56.601851 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d6c4d95fb-tvs2b_da44b8e5-2f5a-446b-9aa3-f367dc008d81/console/0.log" Sep 11 14:38:56.651160 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:56.651132 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-n29f8_406c7df7-aeda-48a9-8663-f87d01d2b033/download-server/0.log" Sep 11 14:38:57.361267 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:57.361234 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-5ghjk_567799ac-c5ee-4fbd-b496-5290693eda3f/volume-data-source-validator/0.log" Sep 11 14:38:58.599223 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:58.599189 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zpsfz_d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8/dns/0.log" Sep 11 14:38:58.622959 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:58.622929 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zpsfz_d6aa3fd6-aab7-4feb-8f3c-3762e7e1c9d8/kube-rbac-proxy/0.log" Sep 11 14:38:58.708159 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:58.708134 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pm6vt_0ffa4002-eab4-4d53-aad1-c82ba3810441/dns-node-resolver/0.log" Sep 11 14:38:59.466771 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:38:59.466726 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mjmwj_c2c22dff-0209-4bda-9a7a-5626d2268974/node-ca/0.log" Sep 11 14:39:00.516667 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:00.516633 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht_4f08071d-1a99-4cd9-8929-f202cd23051d/istio-proxy/0.log" Sep 11 14:39:00.876412 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:00.876325 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8494df7896-tszz7_45ab0a34-6a08-4cc3-9754-549294bd0b74/kube-auth-proxy/0.log" Sep 11 14:39:00.907131 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:00.907079 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-t98fx_31429fc7-4a21-4e5d-9db7-0a0663bf8a71/istio-proxy/0.log" Sep 11 14:39:00.926143 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:00.926111 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-t98fx_31429fc7-4a21-4e5d-9db7-0a0663bf8a71/istio-proxy/1.log" Sep 11 14:39:01.011647 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:01.011614 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-rjfcb_417e7a61-fc2b-4125-97cf-53ef51e93982/payload-processing/0.log" Sep 11 14:39:01.933757 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:01.933727 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gwb7j_08e2fadb-d68b-469d-a7f7-6356a9907e6c/serve-healthcheck-canary/0.log" Sep 11 14:39:02.528338 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:02.528302 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-h9mh7_89a2ef95-796d-4a09-97be-fdaa2d981b30/insights-operator/1.log" Sep 11 14:39:02.528580 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:02.528564 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-h9mh7_89a2ef95-796d-4a09-97be-fdaa2d981b30/insights-operator/0.log" Sep 11 14:39:02.741030 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:02.741002 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vkpjz_ba31e115-ed2c-4929-8c37-d4342d62147f/kube-rbac-proxy/0.log" Sep 11 14:39:02.765075 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:02.765037 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vkpjz_ba31e115-ed2c-4929-8c37-d4342d62147f/exporter/0.log" Sep 11 14:39:02.797613 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:02.797529 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vkpjz_ba31e115-ed2c-4929-8c37-d4342d62147f/extractor/0.log" Sep 11 14:39:03.456354 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:03.456324 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ktch4/perf-node-gather-daemonset-27pqh" Sep 11 14:39:05.278427 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:05.278396 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jxzrs_2b4bed51-642c-46d6-a9f7-a1184b6d21be/manager/0.log" Sep 11 14:39:05.619466 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:05.619434 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vwzkb_922f7c36-5337-45e2-a7b6-94f1ced90673/manager/0.log" Sep 11 14:39:06.030901 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:06.030870 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586bcdcf6-z2w7n_b8119fc8-fece-463f-8abb-c549cbbb0126/manager/0.log" Sep 11 14:39:06.049471 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:06.049438 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5586bcdcf6-z2w7n_b8119fc8-fece-463f-8abb-c549cbbb0126/copy-manifests/0.log" Sep 11 14:39:06.850678 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:06.850643 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht_4f08071d-1a99-4cd9-8929-f202cd23051d/istio-proxy/0.log" Sep 11 14:39:07.215682 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:07.215594 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8494df7896-tszz7_45ab0a34-6a08-4cc3-9754-549294bd0b74/kube-auth-proxy/0.log" Sep 11 14:39:07.237191 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:07.237147 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-t98fx_31429fc7-4a21-4e5d-9db7-0a0663bf8a71/istio-proxy/0.log" Sep 11 14:39:07.253246 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:07.253216 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-t98fx_31429fc7-4a21-4e5d-9db7-0a0663bf8a71/istio-proxy/1.log" Sep 11 14:39:07.337817 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:07.337395 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-rjfcb_417e7a61-fc2b-4125-97cf-53ef51e93982/payload-processing/0.log" Sep 11 14:39:08.837055 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:08.837020 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zkwht_4f08071d-1a99-4cd9-8929-f202cd23051d/istio-proxy/0.log" Sep 11 14:39:09.191193 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:09.191116 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8494df7896-tszz7_45ab0a34-6a08-4cc3-9754-549294bd0b74/kube-auth-proxy/0.log" Sep 11 14:39:09.218062 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:09.218030 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-t98fx_31429fc7-4a21-4e5d-9db7-0a0663bf8a71/istio-proxy/0.log" Sep 11 14:39:09.236517 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:09.236488 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-t98fx_31429fc7-4a21-4e5d-9db7-0a0663bf8a71/istio-proxy/1.log" Sep 11 14:39:09.307415 ip-10-0-136-92 kubenswrapper[2566]: I0911 14:39:09.307387 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-rjfcb_417e7a61-fc2b-4125-97cf-53ef51e93982/payload-processing/0.log"