Sep 08 13:14:31.145575 ip-10-0-134-224 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 13:14:31.145588 ip-10-0-134-224 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 13:14:31.145596 ip-10-0-134-224 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 13:14:31.145886 ip-10-0-134-224 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 13:14:40.255785 ip-10-0-134-224 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 08 13:14:40.255797 ip-10-0-134-224 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot df22b57875c44e869cc48d2e6578af9b --
Sep 08 13:16:21.030142 ip-10-0-134-224 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 13:16:21.450095 ip-10-0-134-224 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:21.450095 ip-10-0-134-224 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 13:16:21.450095 ip-10-0-134-224 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:21.450095 ip-10-0-134-224 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 13:16:21.450095 ip-10-0-134-224 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:21.450858 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.450120    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 13:16:21.454323 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454300    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:21.454323 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454319    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:21.454323 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454325    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:21.454323 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454330    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454336    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454341    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454345    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454349    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454353    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454357    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454361    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454370    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454375    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454378    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454382    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454386    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454390    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454394    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454398    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454402    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454406    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454411    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454415    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:21.454586 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454419    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454423    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454427    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454431    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454435    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454440    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454445    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454449    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454453    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454458    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454462    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454467    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454472    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454476    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454481    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454485    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454489    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454494    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454498    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454502    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:21.455280 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454507    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454513    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454517    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454522    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454526    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454530    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454535    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454539    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454544    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454550    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454555    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454559    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454563    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454568    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454572    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454579    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454585    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454592    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454598    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:21.455813 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454603    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454608    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454613    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454617    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454624    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454629    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454634    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454639    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454644    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454649    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454653    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454660    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454665    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454669    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454674    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454679    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454684    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454688    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454692    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454697    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:21.456309 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454701    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454705    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454712    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454718    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454723    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454728    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454733    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454738    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454742    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454747    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454752    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454756    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454761    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454766    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454770    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454774    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:21.456802 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.454781    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:21.458730 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458677    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:21.458897 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458755    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:21.458943 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458904    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:21.458943 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458909    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:21.458943 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458912    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:21.458943 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458916    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:21.458943 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458919    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:21.458943 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458923    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:21.458943 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458927    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:21.458943 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458929    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:21.458943 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458932    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:21.458943 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458937    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:21.458943 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458942    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:21.458943 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458946    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:21.458943 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458949    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458952    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458956    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458959    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458962    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458965    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458968    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458971    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458974    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458976    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458979    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458983    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458985    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458988    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458991    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458993    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458996    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.458998    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459001    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459003    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:21.459282 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459006    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459008    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459011    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459013    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459016    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459019    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459023    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459027    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459030    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459033    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459036    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459039    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459041    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459044    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459047    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459050    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459053    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459055    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459058    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459063    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:21.459767 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459066    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459069    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459072    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459075    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459078    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459081    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459083    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459086    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459089    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459092    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459095    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459100    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459104    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459106    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459109    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459112    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459115    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459118    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459121    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:21.460272 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459123    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459126    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459129    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459131    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459134    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459137    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459140    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459142    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459145    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459148    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459150    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459153    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459155    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459158    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459161    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459164    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459167    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459169    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459172    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459174    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:21.460737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459177    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459179    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459182    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459184    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459187    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459190    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459565    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459571    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459575    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459578    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459581    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459584    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459587    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459590    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459593    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459595    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459598    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459601    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459604    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:21.461268 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459609    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459612    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459615    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459618    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459621    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459624    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459626    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459629    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459631    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459634    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459636    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459639    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459642    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459644    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459647    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459650    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459653    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459655    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459658    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459660    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:21.461737 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459663    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459666    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459669    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459671    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459674    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459677    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459679    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459682    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459684    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459687    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459690    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459692    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459697    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459700    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459702    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459705    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459709    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459713    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459716    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:21.462234 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459718    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459721    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459723    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459726    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459728    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459732    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459735    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459738    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459742    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459745    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459748    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459751    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459754    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459756    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459759    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459762    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459764    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459767    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459770    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459772    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:21.462741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459775    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459778    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459781    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459784    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459786    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459790    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459793    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459795    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459798    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459801    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459805    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459807    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459810    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459813    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459815    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459818    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459820    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459823    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459826    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459829    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:21.463239 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459831    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459834    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459836    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459839    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459842    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459844    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.459847    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.459947    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.459985    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.459991    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.459996    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460000    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460003    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460007    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460011    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460014    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460017    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460022    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460025    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460028    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460031    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460034    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460037    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 08 13:16:21.463770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460041    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460044    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460048    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460051    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460054    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460057    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460060    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460064    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460067    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460070    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460073    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460076    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460079    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460082    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460085    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460088    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460092    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460095    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460098    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460101    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460104    2578 flags.go:64] FLAG: --enable-server="true"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460107    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460111    2578 flags.go:64] FLAG: --event-burst="100"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460114    2578 flags.go:64] FLAG: --event-qps="50"
Sep 08 13:16:21.464336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460117    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460120    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460125    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460129    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460133    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460136    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460139    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460142    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460145    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460148    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460151    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460154    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460157    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460159    2578 flags.go:64] FLAG: --feature-gates=""
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460165    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460168    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460171    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460174    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460177    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460180    2578 flags.go:64] FLAG: --help="false"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460183    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-134-224.ec2.internal"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460187    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460190    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 13:16:21.464934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460193    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460196    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460199    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460202    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460205    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460207    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460210    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460213    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460216    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460219    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460222    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460226    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460229    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460232    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460235    2578 flags.go:64] FLAG: --lock-file=""
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460238    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460241    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460243    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460249    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460253    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460256    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460258    2578 flags.go:64] FLAG: --logging-format="text"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460261    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460266    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 13:16:21.465514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460269    2578 flags.go:64] FLAG: --manifest-url=""
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460271    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460276    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460279    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460283    2578 flags.go:64] FLAG: --max-pods="110"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460286    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460289    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460292    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460295    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460298    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460301    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460304    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460311    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460315    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460317    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460320    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460324    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460328    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460331    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460335    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460339    2578 flags.go:64] FLAG: --port="10250"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460342    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460345    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d94a45e154f05501"
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460348    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 08 13:16:21.466125 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460352    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460354    2578 flags.go:64] FLAG: --register-node="true"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460357    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460361    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460364    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460367    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460370    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460375    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460378    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460381    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460384    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460387    2578 flags.go:64] FLAG: --runonce="false"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460390    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460393    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460396    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460399    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460402    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460405    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460408    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460411    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460414    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460417    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460420    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460423    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460426    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 13:16:21.466709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460428    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460431    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460437    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460441    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460444    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460448    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460451    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460453    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460456    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460459    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460462    2578 flags.go:64] FLAG: --v="2"
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460465    2578 flags.go:64] FLAG: --version="false"
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460470    2578 flags.go:64] FLAG: --vmodule=""
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460473    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.460478    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460589    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460593    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460596    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460598    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460601    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460603    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460606    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460609    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:21.467376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460611    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460614    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460616    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460619    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460621    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460624    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460627    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460630    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460632    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460635    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460638    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460640    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460644    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460647    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460650    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460652    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460655    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460657    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460660    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460663    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:21.467968 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460665    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460668    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460671    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460674    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460679    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460682    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460685    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460688    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460691    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460694    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460697    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460700    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460703    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460706    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460708    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460711    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460713    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460716    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460718    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:21.468466 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460721    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460723    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460726    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460730    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460733    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460738    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460741    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460744    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460746    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460749    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460752    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460754    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460757    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460759    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460762    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460764    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460789    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460793    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460796    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:21.468954 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460798    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460801    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460804    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460807    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460810    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460812    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460815    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460817    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460822    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460825    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460828    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460831    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460834    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460837    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460840    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460842    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460845    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460848    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460852    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460854    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:21.469438 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460857    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:21.469950 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460873    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:21.469950 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460876    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:21.469950 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460879    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:21.469950 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460881    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:21.469950 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460884    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:21.469950 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460887    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:21.469950 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460889    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:21.469950 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460892    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:21.469950 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460896    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:21.469950 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460899    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:21.469950 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460902    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:21.469950 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.460905    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:21.469950 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.461568    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:21.469950 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.469248    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 13:16:21.469950 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.469264    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.469277    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.469282    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469336    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469341    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469345    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469348    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469352    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469356    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469359    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469362    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469365    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469368    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469370    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469373    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469376    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469379    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469381    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469384    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:21.470314 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469387    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469389    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469392    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469394    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469397    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469400    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469402    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469405    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469414    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469417    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469420    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469423    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469425    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469429    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469432    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469434    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469437    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469440    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469444    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469448    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:21.470858 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469451    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469455    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469458    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469461    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469464    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469466    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469470    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469473    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469476    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469478    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469481    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469484    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469487    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469490    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469493    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469496    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469499    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469501    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469504    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469506    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:21.471368 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469509    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469513    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469515    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469518    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469520    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469524    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469527    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469529    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469532    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469535    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469537    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469540    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469542    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469545    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469548    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469550    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469553    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469555    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469558    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469560    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:21.471912 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469563    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469566    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469568    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469571    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469573    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469576    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469578    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469581    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469583    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469586    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469588    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469590    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469594    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469598    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469601    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469603    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469606    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469609    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469613    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:21.472510 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469615    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:21.473012 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469618    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:21.473012 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469620    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:21.473012 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469623    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:21.473012 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.469628    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:21.473012 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469734    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:21.473012 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469739    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:21.473012 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469741    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:21.473012 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469745    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:21.473012 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469748    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:21.473012 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469751    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:21.473012 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469753    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:21.473012 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469756    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:21.473012 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469759    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:21.473012 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469762    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:21.473012 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469764    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469767    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469769    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469772    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469774    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469777    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469779    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469782    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469785    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469787    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469790    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469792    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469795    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469798    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469800    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469803    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469806    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469810    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469813    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:21.473391 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469816    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469819    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469821    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469824    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469827    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469829    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469832    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469835    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469838    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469841    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469843    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469846    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469848    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469851    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469853    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469856    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469858    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469878    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469881    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:21.473853 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469884    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469886    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469889    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469892    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469894    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469897    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469899    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469902    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469905    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469908    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469911    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469913    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469917    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469921    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469924    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469927    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469930    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469933    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469937    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469940    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:21.474336 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469943    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469946    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469949    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469951    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469954    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469957    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469959    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469962    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469965    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469967    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469970    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469973    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469975    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469977    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469980    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469983    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469985    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469988    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469990    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469993    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:21.474830 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469995    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:21.475376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.469998    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:21.475376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.470001    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:21.475376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.470004    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:21.475376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.470006    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:21.475376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.470009    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:21.475376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.470012    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:21.475376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.470015    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:21.475376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.470017    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:21.475376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.470020    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:21.475376 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:21.470022    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:21.475376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.470027    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:21.475376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.470132    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 13:16:21.475376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.473490    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 13:16:21.475376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.474614    2578 server.go:1019] "Starting client certificate rotation"
Sep 08 13:16:21.475719 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.474707    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:16:21.475719 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.474746    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:16:21.477448 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.477434    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:16:21.478555 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.478538    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:16:21.492240 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.492223    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 08 13:16:21.495844 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.495825    2578 log.go:25] "Validated CRI v1 image API"
Sep 08 13:16:21.497592 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.497550    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 13:16:21.507021 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.507001    2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d9059326-1c77-4379-81b3-3d4ada77ced9:/dev/nvme0n1p4 fcc4a841-64c9-46ca-9771-a88d91c8b00e:/dev/nvme0n1p3]
Sep 08 13:16:21.507094 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.507019    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 13:16:21.507935 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.507917    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:16:21.512916 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.512798    2578 manager.go:216] Machine: {Timestamp:2026-09-08 13:16:21.511668685 +0000 UTC m=+0.375522241 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099396 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27524382da77414c16aae197e7feb1 SystemUUID:ec275243-82da-7741-4c16-aae197e7feb1 BootID:df22b578-75c4-4e86-9cc4-8d2e6578af9b Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f7:8a:1b:bd:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f7:8a:1b:bd:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:3d:92:bf:53:0b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 13:16:21.512916 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.512911    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 13:16:21.513044 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.512986    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 13:16:21.514179 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.514147    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 13:16:21.514308 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.514182    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-224.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 13:16:21.515073 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.515063    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 13:16:21.515112 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.515076    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 13:16:21.515112 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.515084    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:16:21.516057 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.516047    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:16:21.517944 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.517934    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 13:16:21.523280 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.523269    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:16:21.523397 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.523387    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 13:16:21.523428 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.523416    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 13:16:21.523457 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.523433    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 13:16:21.523457 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.523451    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 13:16:21.523541 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.523461    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 08 13:16:21.523541 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.523469    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 13:16:21.523797 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.523786    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:16:21.523842 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.523804    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:16:21.526141 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.526125    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 13:16:21.528969 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.528953    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 13:16:21.529027 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.528996    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 13:16:21.529169 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.529160    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 13:16:21.529207 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.529174    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 13:16:21.529207 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.529180    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 13:16:21.529207 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.529186    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 13:16:21.529207 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.529193    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 13:16:21.529207 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.529198    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 13:16:21.529818 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.529810    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 13:16:21.529850 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.529818    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 13:16:21.529850 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.529825    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 13:16:21.529850 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.529830    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 13:16:21.529850 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.529840    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 13:16:21.529850 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.529848    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 13:16:21.530824 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.530813    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 13:16:21.530858 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.530824    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 13:16:21.533602 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.533583    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 13:16:21.533873 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.533841    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-224.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 13:16:21.534074 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.533951    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-224.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 13:16:21.534685 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.534674    2578 server.go:1268] "Started kubelet"
Sep 08 13:16:21.534763 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.534740    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 13:16:21.534883 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.534827    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 13:16:21.535015 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.534901    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 13:16:21.536370 ip-10-0-134-224 systemd[1]: Started Kubernetes Kubelet.
Sep 08 13:16:21.537293 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.537273    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 13:16:21.539030 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.538349    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 13:16:21.541582 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.541565    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-79ctq"
Sep 08 13:16:21.542634 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.541489    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-224.ec2.internal.18d35adcd6862e13  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-224.ec2.internal,UID:ip-10-0-134-224.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-224.ec2.internal,},FirstTimestamp:2026-09-08 13:16:21.534641683 +0000 UTC m=+0.398495239,LastTimestamp:2026-09-08 13:16:21.534641683 +0000 UTC m=+0.398495239,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-224.ec2.internal,}"
Sep 08 13:16:21.543029 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.543008    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 13:16:21.543260 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.543242    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 13:16:21.543353 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.543248    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 13:16:21.543421 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.543354    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:21.544224 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.544103    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 13:16:21.544304 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.544229    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 13:16:21.544399 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.544386    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 13:16:21.544454 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.544400    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 13:16:21.544529 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.544509    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 13:16:21.545151 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.545123    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 13:16:21.545151 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.545141    2578 factory.go:55] Registering systemd factory
Sep 08 13:16:21.545151 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.545151    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 08 13:16:21.545448 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.545430    2578 factory.go:153] Registering CRI-O factory
Sep 08 13:16:21.545527 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.545453    2578 factory.go:223] Registration of the crio container factory successfully
Sep 08 13:16:21.545527 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.545473    2578 factory.go:103] Registering Raw factory
Sep 08 13:16:21.545527 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.545495    2578 manager.go:1196] Started watching for new ooms in manager
Sep 08 13:16:21.545907 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.545890    2578 manager.go:349] Starting recovery of all containers
Sep 08 13:16:21.547887 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.547847    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-79ctq"
Sep 08 13:16:21.549498 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.549384    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 13:16:21.549498 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.549488    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-224.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 13:16:21.556946 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.556921    2578 manager.go:354] Recovery completed
Sep 08 13:16:21.561480 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.561463    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:21.563151 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.563137    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:21.563221 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.563174    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:21.563221 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.563189    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:21.563754 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.563740    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 13:16:21.563754 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.563753    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 13:16:21.563848 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.563769    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:16:21.564775 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.564761    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:21.565941 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.565929    2578 policy_none.go:49] "None policy: Start"
Sep 08 13:16:21.565999 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.565948    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 13:16:21.565999 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.565960    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 13:16:21.566934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.566918    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:21.567004 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.566949    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:21.567004 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.566965    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:21.569193 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.569180    2578 policy_none.go:47] "Start"
Sep 08 13:16:21.596625 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.596609    2578 manager.go:337] "Starting Device Plugin manager"
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.596652    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.596666    2578 server.go:87] "Starting device plugin registration server"
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.597009    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.597020    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.597127    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.597223    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.597236    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.597895    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.597944    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.697474    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.698318    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.698357    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.698371    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.698422    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-224.ec2.internal"
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.707146    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-224.ec2.internal"
Sep 08 13:16:21.711633 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.707174    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-224.ec2.internal\": node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:21.722282 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.722258    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:21.742339 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.742311    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 13:16:21.743435 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.743420    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 13:16:21.743508 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.743466    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 13:16:21.743508 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.743485    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 13:16:21.743609 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.743522    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 13:16:21.745607 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.745588    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:21.823164 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.823146    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:21.844337 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.844289    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-224.ec2.internal"]
Sep 08 13:16:21.844427 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.844389    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:21.845665 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.845644    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/868db1574818d2dd017de1d6b08036fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal\" (UID: \"868db1574818d2dd017de1d6b08036fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal"
Sep 08 13:16:21.845745 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.845678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/868db1574818d2dd017de1d6b08036fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal\" (UID: \"868db1574818d2dd017de1d6b08036fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal"
Sep 08 13:16:21.845745 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.845733    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:21.845824 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.845755    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:21.845824 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.845767    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:21.849080 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.849062    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:21.849194 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.849176    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal"
Sep 08 13:16:21.849240 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.849215    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:21.849790 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.849774    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:21.849790 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.849784    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:21.849919 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.849802    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:21.849919 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.849814    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:21.849919 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.849803    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:21.849919 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.849909    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:21.851979 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.851966    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.ec2.internal"
Sep 08 13:16:21.852031 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.851989    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:21.852663 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.852645    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:21.852738 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.852680    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:21.852738 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.852690    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:21.878404 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.878380    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-224.ec2.internal\" not found" node="ip-10-0-134-224.ec2.internal"
Sep 08 13:16:21.881679 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.881664    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-224.ec2.internal\" not found" node="ip-10-0-134-224.ec2.internal"
Sep 08 13:16:21.923921 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:21.923901    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:21.946281 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.946257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/868db1574818d2dd017de1d6b08036fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal\" (UID: \"868db1574818d2dd017de1d6b08036fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal"
Sep 08 13:16:21.946372 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.946289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/868db1574818d2dd017de1d6b08036fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal\" (UID: \"868db1574818d2dd017de1d6b08036fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal"
Sep 08 13:16:21.946372 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.946308    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bed360bd3f4f5cc25abd3af9aed5d2dd-config\") pod \"kube-apiserver-proxy-ip-10-0-134-224.ec2.internal\" (UID: \"bed360bd3f4f5cc25abd3af9aed5d2dd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.ec2.internal"
Sep 08 13:16:21.946372 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.946365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/868db1574818d2dd017de1d6b08036fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal\" (UID: \"868db1574818d2dd017de1d6b08036fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal"
Sep 08 13:16:21.946468 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:21.946370    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/868db1574818d2dd017de1d6b08036fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal\" (UID: \"868db1574818d2dd017de1d6b08036fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal"
Sep 08 13:16:22.024575 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:22.024519    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:22.046993 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:22.046960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bed360bd3f4f5cc25abd3af9aed5d2dd-config\") pod \"kube-apiserver-proxy-ip-10-0-134-224.ec2.internal\" (UID: \"bed360bd3f4f5cc25abd3af9aed5d2dd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.ec2.internal"
Sep 08 13:16:22.047065 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:22.047020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bed360bd3f4f5cc25abd3af9aed5d2dd-config\") pod \"kube-apiserver-proxy-ip-10-0-134-224.ec2.internal\" (UID: \"bed360bd3f4f5cc25abd3af9aed5d2dd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.ec2.internal"
Sep 08 13:16:22.125138 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:22.125112    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:22.180642 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:22.180611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal"
Sep 08 13:16:22.183767 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:22.183754    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.ec2.internal"
Sep 08 13:16:22.225354 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:22.225326    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:23.589691 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:23.589663    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:23.690223 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:23.690189    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:23.737727 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:23.737693    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 13:16:23.738147 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:23.737846    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 13:16:23.791166 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:23.791140    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:23.814764 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:23.814696    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 13:11:22 +0000 UTC" deadline="2028-04-03 10:30:58.689138128 +0000 UTC"
Sep 08 13:16:23.814764 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:23.814724    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13749h14m34.874417077s"
Sep 08 13:16:23.891771 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:23.891748    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:23.992232 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:23.992197    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:24.005092 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.005071    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:24.092498 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:24.092437    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:24.193003 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:24.192969    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-224.ec2.internal\" not found"
Sep 08 13:16:24.217612 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.217594    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:24.307427 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.307403    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal"
Sep 08 13:16:24.325946 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.325929    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:16:24.326695 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.326683    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.ec2.internal"
Sep 08 13:16:24.336452 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.336437    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:16:24.338602 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.338585    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:24.789104 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.789079    2578 apiserver.go:52] "Watching apiserver"
Sep 08 13:16:24.799056 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.799037    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 13:16:24.799409 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.799387    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-224.ec2.internal","openshift-dns/node-resolver-j9nqf","openshift-image-registry/node-ca-kp8g5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal","openshift-multus/multus-2jfd4","openshift-multus/network-metrics-daemon-h2td8","openshift-network-diagnostics/network-check-target-csgtz","openshift-ovn-kubernetes/ovnkube-node-ms6p6","kube-system/konnectivity-agent-p6sbc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8","openshift-cluster-node-tuning-operator/tuned-kzlgx","openshift-multus/multus-additional-cni-plugins-95zgl","openshift-network-operator/iptables-alerter-kn6cq"]
Sep 08 13:16:24.801656 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.801639    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.803718 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.803698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j9nqf"
Sep 08 13:16:24.805267 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.805252    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 13:16:24.805464 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.805451    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 13:16:24.805808 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.805792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kp8g5"
Sep 08 13:16:24.805912 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.805833    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 13:16:24.805912 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.805853    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vvd4m\""
Sep 08 13:16:24.805912 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.805868    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 13:16:24.806207 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.806192    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 13:16:24.806737 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.806721    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vwvxm\""
Sep 08 13:16:24.806860 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.806766    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 13:16:24.807827 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.807808    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 13:16:24.807953 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.807935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:24.808411 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:24.808376    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:24.808666 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.808541    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 13:16:24.808753 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.808699    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 13:16:24.810564 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.809360    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 13:16:24.810564 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.809362    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zw9kn\""
Sep 08 13:16:24.811858 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.810713    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.813372 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.813352    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 13:16:24.813672 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.813521    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:24.813672 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:24.813590    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:24.813979 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.813954    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 13:16:24.814170 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.814153    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2qng6\""
Sep 08 13:16:24.814770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.814359    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 13:16:24.815794 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.815770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-sys-fs\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.815894 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.815801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-cnibin\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.815894 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.815818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-var-lib-kubelet\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.815894 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.815840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-multus-conf-dir\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.815894 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.815861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-etc-kubernetes\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.816102 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.815899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58zsj\" (UniqueName: \"kubernetes.io/projected/1ba25c2e-e201-4764-adb2-e8faa0761d29-kube-api-access-58zsj\") pod \"node-resolver-j9nqf\" (UID: \"1ba25c2e-e201-4764-adb2-e8faa0761d29\") " pod="openshift-dns/node-resolver-j9nqf"
Sep 08 13:16:24.816102 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.815976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.816102 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-run-netns\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.816102 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-run-multus-certs\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.816102 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba25c2e-e201-4764-adb2-e8faa0761d29-tmp-dir\") pod \"node-resolver-j9nqf\" (UID: \"1ba25c2e-e201-4764-adb2-e8faa0761d29\") " pod="openshift-dns/node-resolver-j9nqf"
Sep 08 13:16:24.816102 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-multus-cni-dir\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.816102 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-var-lib-cni-bin\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.816429 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-hostroot\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.816429 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816149    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-multus-socket-dir-parent\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.816429 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816187    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-var-lib-cni-multus\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.816429 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d7a807ed-4b45-45f2-a302-123e9b392de3-multus-daemon-config\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.816429 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816256    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.816429 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-socket-dir\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.816429 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816290    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkh6\" (UniqueName: \"kubernetes.io/projected/d7a807ed-4b45-45f2-a302-123e9b392de3-kube-api-access-dbkh6\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.816429 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-registration-dir\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.816429 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-device-dir\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.816429 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7a807ed-4b45-45f2-a302-123e9b392de3-cni-binary-copy\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.816429 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-run-k8s-cni-cncf-io\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.816429 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.817051 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tsd8\" (UniqueName: \"kubernetes.io/projected/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-kube-api-access-5tsd8\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.817051 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-system-cni-dir\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.817051 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816482    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ba25c2e-e201-4764-adb2-e8faa0761d29-hosts-file\") pod \"node-resolver-j9nqf\" (UID: \"1ba25c2e-e201-4764-adb2-e8faa0761d29\") " pod="openshift-dns/node-resolver-j9nqf"
Sep 08 13:16:24.817051 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816496    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0f546ae-e7b5-4459-a35a-c16a0e7807b9-host\") pod \"node-ca-kp8g5\" (UID: \"a0f546ae-e7b5-4459-a35a-c16a0e7807b9\") " pod="openshift-image-registry/node-ca-kp8g5"
Sep 08 13:16:24.817051 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2pxt\" (UniqueName: \"kubernetes.io/projected/a0f546ae-e7b5-4459-a35a-c16a0e7807b9-kube-api-access-s2pxt\") pod \"node-ca-kp8g5\" (UID: \"a0f546ae-e7b5-4459-a35a-c16a0e7807b9\") " pod="openshift-image-registry/node-ca-kp8g5"
Sep 08 13:16:24.817051 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816524    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs\") pod \"network-metrics-daemon-h2td8\" (UID: \"7eeefbda-9e69-4275-a756-6effb7738d90\") " pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:24.817051 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a0f546ae-e7b5-4459-a35a-c16a0e7807b9-serviceca\") pod \"node-ca-kp8g5\" (UID: \"a0f546ae-e7b5-4459-a35a-c16a0e7807b9\") " pod="openshift-image-registry/node-ca-kp8g5"
Sep 08 13:16:24.817051 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.816551    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqxjl\" (UniqueName: \"kubernetes.io/projected/7eeefbda-9e69-4275-a756-6effb7738d90-kube-api-access-zqxjl\") pod \"network-metrics-daemon-h2td8\" (UID: \"7eeefbda-9e69-4275-a756-6effb7738d90\") " pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:24.818703 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.818565    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p6sbc"
Sep 08 13:16:24.818994 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.818975    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 13:16:24.819104 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.819084    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 13:16:24.819513 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.819493    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6svj4\""
Sep 08 13:16:24.819607 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.819545    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 13:16:24.820724 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.820693    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 13:16:24.820724 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.820695    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 13:16:24.820888 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.820733    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 13:16:24.820997 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.820978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:24.822755 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.822728    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 13:16:24.822851 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.822735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 13:16:24.823132 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.823119    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x55z2\""
Sep 08 13:16:24.823339 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.823324    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:24.823791 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.823771    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:16:24.823871 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.823835    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 13:16:24.823918 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.823871    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:16:24.824139 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.824126    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7kqxw\""
Sep 08 13:16:24.825505 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.825487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kn6cq"
Sep 08 13:16:24.827855 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.827733    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 13:16:24.827855 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.827804    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6ggnj\""
Sep 08 13:16:24.827995 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.827957    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 13:16:24.828742 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.828721    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:16:24.828855 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.828804    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 13:16:24.828922 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.828865    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 13:16:24.828922 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.828908    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sdzsd\""
Sep 08 13:16:24.842283 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.842266    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8fjsn"
Sep 08 13:16:24.851230 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.851211    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8fjsn"
Sep 08 13:16:24.908994 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.908976    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 13:16:24.917077 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7a807ed-4b45-45f2-a302-123e9b392de3-cni-binary-copy\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.917152 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-systemd-units\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.917227 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-sys\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:24.917283 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-tuned\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:24.917333 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tsd8\" (UniqueName: \"kubernetes.io/projected/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-kube-api-access-5tsd8\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.917383 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-system-cni-dir\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.917482 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ba25c2e-e201-4764-adb2-e8faa0761d29-hosts-file\") pod \"node-resolver-j9nqf\" (UID: \"1ba25c2e-e201-4764-adb2-e8faa0761d29\") " pod="openshift-dns/node-resolver-j9nqf"
Sep 08 13:16:24.917537 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-system-cni-dir\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.917537 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2pxt\" (UniqueName: \"kubernetes.io/projected/a0f546ae-e7b5-4459-a35a-c16a0e7807b9-kube-api-access-s2pxt\") pod \"node-ca-kp8g5\" (UID: \"a0f546ae-e7b5-4459-a35a-c16a0e7807b9\") " pod="openshift-image-registry/node-ca-kp8g5"
Sep 08 13:16:24.917537 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ba25c2e-e201-4764-adb2-e8faa0761d29-hosts-file\") pod \"node-resolver-j9nqf\" (UID: \"1ba25c2e-e201-4764-adb2-e8faa0761d29\") " pod="openshift-dns/node-resolver-j9nqf"
Sep 08 13:16:24.917537 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7a807ed-4b45-45f2-a302-123e9b392de3-cni-binary-copy\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.917751 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghm85\" (UniqueName: \"kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85\") pod \"network-check-target-csgtz\" (UID: \"3f818ca3-1324-41d6-9d78-46c09e9719a3\") " pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:24.917751 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b8f2f57-9508-463c-b76b-14b6207309fb-host-slash\") pod \"iptables-alerter-kn6cq\" (UID: \"9b8f2f57-9508-463c-b76b-14b6207309fb\") " pod="openshift-network-operator/iptables-alerter-kn6cq"
Sep 08 13:16:24.917751 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-kubernetes\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:24.917751 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-run\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:24.917751 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a0f546ae-e7b5-4459-a35a-c16a0e7807b9-serviceca\") pod \"node-ca-kp8g5\" (UID: \"a0f546ae-e7b5-4459-a35a-c16a0e7807b9\") " pod="openshift-image-registry/node-ca-kp8g5"
Sep 08 13:16:24.917751 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-var-lib-kubelet\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:24.917751 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-host\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:24.917751 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-sys-fs\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.918035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-cnibin\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.918035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-etc-kubernetes\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.918035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-etc-kubernetes\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.918035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917856    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-sys-fs\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.918035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-cnibin\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.918035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wznpm\" (UniqueName: \"kubernetes.io/projected/9b8f2f57-9508-463c-b76b-14b6207309fb-kube-api-access-wznpm\") pod \"iptables-alerter-kn6cq\" (UID: \"9b8f2f57-9508-463c-b76b-14b6207309fb\") " pod="openshift-network-operator/iptables-alerter-kn6cq"
Sep 08 13:16:24.918035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e40e15f-a782-46d6-b275-7722e86f4652-agent-certs\") pod \"konnectivity-agent-p6sbc\" (UID: \"4e40e15f-a782-46d6-b275-7722e86f4652\") " pod="kube-system/konnectivity-agent-p6sbc"
Sep 08 13:16:24.918035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75a311a2-7bf2-4973-8db8-a4c47baf06e7-tmp\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:24.918035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-var-lib-cni-bin\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.918035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-hostroot\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.918035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.917992    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a0f546ae-e7b5-4459-a35a-c16a0e7807b9-serviceca\") pod \"node-ca-kp8g5\" (UID: \"a0f546ae-e7b5-4459-a35a-c16a0e7807b9\") " pod="openshift-image-registry/node-ca-kp8g5"
Sep 08 13:16:24.918035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-run-netns\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.918035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-run-systemd\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918043    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-var-lib-cni-bin\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918054    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-hostroot\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xjm9\" (UniqueName: \"kubernetes.io/projected/d609c6d5-329d-4100-951b-96638309f6cc-kube-api-access-9xjm9\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-multus-socket-dir-parent\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-var-lib-cni-multus\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13d85014-a3c5-4627-80f6-62de2e79b1cf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918149    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-multus-socket-dir-parent\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbs99\" (UniqueName: \"kubernetes.io/projected/13d85014-a3c5-4627-80f6-62de2e79b1cf-kube-api-access-kbs99\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-var-lib-cni-multus\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-socket-dir\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-sysctl-conf\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918244    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-socket-dir\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-systemd\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918272    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkh6\" (UniqueName: \"kubernetes.io/projected/d7a807ed-4b45-45f2-a302-123e9b392de3-kube-api-access-dbkh6\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-registration-dir\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.918476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-device-dir\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-registration-dir\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-run-k8s-cni-cncf-io\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-device-dir\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13d85014-a3c5-4627-80f6-62de2e79b1cf-cni-binary-copy\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918470    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-run-k8s-cni-cncf-io\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-cni-netd\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-modprobe-d\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0f546ae-e7b5-4459-a35a-c16a0e7807b9-host\") pod \"node-ca-kp8g5\" (UID: \"a0f546ae-e7b5-4459-a35a-c16a0e7807b9\") " pod="openshift-image-registry/node-ca-kp8g5"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918634    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0f546ae-e7b5-4459-a35a-c16a0e7807b9-host\") pod \"node-ca-kp8g5\" (UID: \"a0f546ae-e7b5-4459-a35a-c16a0e7807b9\") " pod="openshift-image-registry/node-ca-kp8g5"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918662    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e40e15f-a782-46d6-b275-7722e86f4652-konnectivity-ca\") pod \"konnectivity-agent-p6sbc\" (UID: \"4e40e15f-a782-46d6-b275-7722e86f4652\") " pod="kube-system/konnectivity-agent-p6sbc"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918678    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-lib-modules\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs\") pod \"network-metrics-daemon-h2td8\" (UID: \"7eeefbda-9e69-4275-a756-6effb7738d90\") " pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13d85014-a3c5-4627-80f6-62de2e79b1cf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918722    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-etc-openvswitch\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.919328 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-cni-bin\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:24.918814    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:24.918872    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs podName:7eeefbda-9e69-4275-a756-6effb7738d90 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:25.418856337 +0000 UTC m=+3.018953969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs") pod "network-metrics-daemon-h2td8" (UID: "7eeefbda-9e69-4275-a756-6effb7738d90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d609c6d5-329d-4100-951b-96638309f6cc-env-overrides\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqxjl\" (UniqueName: \"kubernetes.io/projected/7eeefbda-9e69-4275-a756-6effb7738d90-kube-api-access-zqxjl\") pod \"network-metrics-daemon-h2td8\" (UID: \"7eeefbda-9e69-4275-a756-6effb7738d90\") " pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-var-lib-kubelet\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-multus-conf-dir\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918976    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-multus-conf-dir\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.918979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-var-lib-kubelet\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919011    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58zsj\" (UniqueName: \"kubernetes.io/projected/1ba25c2e-e201-4764-adb2-e8faa0761d29-kube-api-access-58zsj\") pod \"node-resolver-j9nqf\" (UID: \"1ba25c2e-e201-4764-adb2-e8faa0761d29\") " pod="openshift-dns/node-resolver-j9nqf"
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b8f2f57-9508-463c-b76b-14b6207309fb-iptables-alerter-script\") pod \"iptables-alerter-kn6cq\" (UID: \"9b8f2f57-9508-463c-b76b-14b6207309fb\") " pod="openshift-network-operator/iptables-alerter-kn6cq"
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-kubelet\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-run-ovn\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d609c6d5-329d-4100-951b-96638309f6cc-ovnkube-config\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-run-netns\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.919928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-run-multus-certs\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919204    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba25c2e-e201-4764-adb2-e8faa0761d29-tmp-dir\") pod \"node-resolver-j9nqf\" (UID: \"1ba25c2e-e201-4764-adb2-e8faa0761d29\") " pod="openshift-dns/node-resolver-j9nqf"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13d85014-a3c5-4627-80f6-62de2e79b1cf-system-cni-dir\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-run-netns\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-host-run-multus-certs\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13d85014-a3c5-4627-80f6-62de2e79b1cf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-slash\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-var-lib-openvswitch\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-multus-cni-dir\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-run-openvswitch\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-log-socket\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d609c6d5-329d-4100-951b-96638309f6cc-ovnkube-script-lib\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7a807ed-4b45-45f2-a302-123e9b392de3-multus-cni-dir\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-sysctl-d\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp9pm\" (UniqueName: \"kubernetes.io/projected/75a311a2-7bf2-4973-8db8-a4c47baf06e7-kube-api-access-zp9pm\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba25c2e-e201-4764-adb2-e8faa0761d29-tmp-dir\") pod \"node-resolver-j9nqf\" (UID: \"1ba25c2e-e201-4764-adb2-e8faa0761d29\") " pod="openshift-dns/node-resolver-j9nqf"
Sep 08 13:16:24.920496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d7a807ed-4b45-45f2-a302-123e9b392de3-multus-daemon-config\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.921200 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-node-log\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.921200 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d609c6d5-329d-4100-951b-96638309f6cc-ovn-node-metrics-cert\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:24.921200 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-sysconfig\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:24.921200 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13d85014-a3c5-4627-80f6-62de2e79b1cf-cnibin\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:24.921200 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.919870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d7a807ed-4b45-45f2-a302-123e9b392de3-multus-daemon-config\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.928510 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.928389    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 13:16:24.929768 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:24.929715    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed360bd3f4f5cc25abd3af9aed5d2dd.slice/crio-fea415471b4389fc4fd68477dbedc3f182028342e13f7a0b6929e8c829c47c0b WatchSource:0}: Error finding container fea415471b4389fc4fd68477dbedc3f182028342e13f7a0b6929e8c829c47c0b: Status 404 returned error can't find the container with id fea415471b4389fc4fd68477dbedc3f182028342e13f7a0b6929e8c829c47c0b
Sep 08 13:16:24.932290 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.932264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkh6\" (UniqueName: \"kubernetes.io/projected/d7a807ed-4b45-45f2-a302-123e9b392de3-kube-api-access-dbkh6\") pod \"multus-2jfd4\" (UID: \"d7a807ed-4b45-45f2-a302-123e9b392de3\") " pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:24.932388 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.932287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58zsj\" (UniqueName: \"kubernetes.io/projected/1ba25c2e-e201-4764-adb2-e8faa0761d29-kube-api-access-58zsj\") pod \"node-resolver-j9nqf\" (UID: \"1ba25c2e-e201-4764-adb2-e8faa0761d29\") " pod="openshift-dns/node-resolver-j9nqf"
Sep 08 13:16:24.932388 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.932330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tsd8\" (UniqueName: \"kubernetes.io/projected/878d9839-5817-4ecc-b9ca-0abb3ae7f1a2-kube-api-access-5tsd8\") pod \"aws-ebs-csi-driver-node-652z8\" (UID: \"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:24.932388 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.932329    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqxjl\" (UniqueName: \"kubernetes.io/projected/7eeefbda-9e69-4275-a756-6effb7738d90-kube-api-access-zqxjl\") pod \"network-metrics-daemon-h2td8\" (UID: \"7eeefbda-9e69-4275-a756-6effb7738d90\") " pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:24.932543 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.932514    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2pxt\" (UniqueName: \"kubernetes.io/projected/a0f546ae-e7b5-4459-a35a-c16a0e7807b9-kube-api-access-s2pxt\") pod \"node-ca-kp8g5\" (UID: \"a0f546ae-e7b5-4459-a35a-c16a0e7807b9\") " pod="openshift-image-registry/node-ca-kp8g5"
Sep 08 13:16:24.934972 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:24.934956    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:16:25.011381 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.011341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.ec2.internal" event={"ID":"bed360bd3f4f5cc25abd3af9aed5d2dd","Type":"ContainerStarted","Data":"fea415471b4389fc4fd68477dbedc3f182028342e13f7a0b6929e8c829c47c0b"}
Sep 08 13:16:25.020006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.019983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13d85014-a3c5-4627-80f6-62de2e79b1cf-cnibin\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:25.020103 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-systemd-units\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.020103 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-sys\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.020103 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-tuned\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.020103 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghm85\" (UniqueName: \"kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85\") pod \"network-check-target-csgtz\" (UID: \"3f818ca3-1324-41d6-9d78-46c09e9719a3\") " pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:25.020103 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13d85014-a3c5-4627-80f6-62de2e79b1cf-cnibin\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-systemd-units\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b8f2f57-9508-463c-b76b-14b6207309fb-host-slash\") pod \"iptables-alerter-kn6cq\" (UID: \"9b8f2f57-9508-463c-b76b-14b6207309fb\") " pod="openshift-network-operator/iptables-alerter-kn6cq"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b8f2f57-9508-463c-b76b-14b6207309fb-host-slash\") pod \"iptables-alerter-kn6cq\" (UID: \"9b8f2f57-9508-463c-b76b-14b6207309fb\") " pod="openshift-network-operator/iptables-alerter-kn6cq"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020126    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-sys\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-kubernetes\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-run\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-var-lib-kubelet\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020219    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-kubernetes\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-host\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wznpm\" (UniqueName: \"kubernetes.io/projected/9b8f2f57-9508-463c-b76b-14b6207309fb-kube-api-access-wznpm\") pod \"iptables-alerter-kn6cq\" (UID: \"9b8f2f57-9508-463c-b76b-14b6207309fb\") " pod="openshift-network-operator/iptables-alerter-kn6cq"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e40e15f-a782-46d6-b275-7722e86f4652-agent-certs\") pod \"konnectivity-agent-p6sbc\" (UID: \"4e40e15f-a782-46d6-b275-7722e86f4652\") " pod="kube-system/konnectivity-agent-p6sbc"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020299    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-run\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-var-lib-kubelet\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75a311a2-7bf2-4973-8db8-a4c47baf06e7-tmp\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020276    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-host\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.020349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-run-netns\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-run-systemd\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020399    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xjm9\" (UniqueName: \"kubernetes.io/projected/d609c6d5-329d-4100-951b-96638309f6cc-kube-api-access-9xjm9\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-run-netns\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13d85014-a3c5-4627-80f6-62de2e79b1cf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbs99\" (UniqueName: \"kubernetes.io/projected/13d85014-a3c5-4627-80f6-62de2e79b1cf-kube-api-access-kbs99\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-run-systemd\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-sysctl-conf\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-systemd\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13d85014-a3c5-4627-80f6-62de2e79b1cf-cni-binary-copy\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020650    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13d85014-a3c5-4627-80f6-62de2e79b1cf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-cni-netd\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-modprobe-d\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020698    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-systemd\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e40e15f-a782-46d6-b275-7722e86f4652-konnectivity-ca\") pod \"konnectivity-agent-p6sbc\" (UID: \"4e40e15f-a782-46d6-b275-7722e86f4652\") " pod="kube-system/konnectivity-agent-p6sbc"
Sep 08 13:16:25.021134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-cni-netd\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-lib-modules\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-modprobe-d\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13d85014-a3c5-4627-80f6-62de2e79b1cf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020826    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-sysctl-conf\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020829    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-lib-modules\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-etc-openvswitch\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020878    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-etc-openvswitch\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-cni-bin\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-cni-bin\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.020969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d609c6d5-329d-4100-951b-96638309f6cc-env-overrides\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b8f2f57-9508-463c-b76b-14b6207309fb-iptables-alerter-script\") pod \"iptables-alerter-kn6cq\" (UID: \"9b8f2f57-9508-463c-b76b-14b6207309fb\") " pod="openshift-network-operator/iptables-alerter-kn6cq"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-kubelet\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-run-ovn\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d609c6d5-329d-4100-951b-96638309f6cc-ovnkube-config\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13d85014-a3c5-4627-80f6-62de2e79b1cf-system-cni-dir\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:25.022006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13d85014-a3c5-4627-80f6-62de2e79b1cf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-slash\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-var-lib-openvswitch\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-run-openvswitch\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-log-socket\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d609c6d5-329d-4100-951b-96638309f6cc-ovnkube-script-lib\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-sysctl-d\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021308    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp9pm\" (UniqueName: \"kubernetes.io/projected/75a311a2-7bf2-4973-8db8-a4c47baf06e7-kube-api-access-zp9pm\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13d85014-a3c5-4627-80f6-62de2e79b1cf-cni-binary-copy\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-node-log\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021358    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13d85014-a3c5-4627-80f6-62de2e79b1cf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d609c6d5-329d-4100-951b-96638309f6cc-ovn-node-metrics-cert\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-node-log\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-sysconfig\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-kubelet\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-sysconfig\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021480    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-run-ovn\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.022849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9b8f2f57-9508-463c-b76b-14b6207309fb-iptables-alerter-script\") pod \"iptables-alerter-kn6cq\" (UID: \"9b8f2f57-9508-463c-b76b-14b6207309fb\") " pod="openshift-network-operator/iptables-alerter-kn6cq"
Sep 08 13:16:25.023367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021693    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-run-openvswitch\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.023367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.021741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13d85014-a3c5-4627-80f6-62de2e79b1cf-system-cni-dir\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:25.023367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.022004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d609c6d5-329d-4100-951b-96638309f6cc-env-overrides\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.023367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.022063    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d609c6d5-329d-4100-951b-96638309f6cc-ovnkube-config\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.023367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.022077    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-log-socket\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.023367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.022184    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13d85014-a3c5-4627-80f6-62de2e79b1cf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:25.023367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.022180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-sysctl-d\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.023367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.022206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d609c6d5-329d-4100-951b-96638309f6cc-ovnkube-script-lib\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.023367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.022255    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-slash\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.023367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.022257    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.023367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.022267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d609c6d5-329d-4100-951b-96638309f6cc-var-lib-openvswitch\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.023367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.022602    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75a311a2-7bf2-4973-8db8-a4c47baf06e7-tmp\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.023367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.022739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e40e15f-a782-46d6-b275-7722e86f4652-konnectivity-ca\") pod \"konnectivity-agent-p6sbc\" (UID: \"4e40e15f-a782-46d6-b275-7722e86f4652\") " pod="kube-system/konnectivity-agent-p6sbc"
Sep 08 13:16:25.023367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.022903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e40e15f-a782-46d6-b275-7722e86f4652-agent-certs\") pod \"konnectivity-agent-p6sbc\" (UID: \"4e40e15f-a782-46d6-b275-7722e86f4652\") " pod="kube-system/konnectivity-agent-p6sbc"
Sep 08 13:16:25.024031 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.023304    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/75a311a2-7bf2-4973-8db8-a4c47baf06e7-etc-tuned\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.024031 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.023840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d609c6d5-329d-4100-951b-96638309f6cc-ovn-node-metrics-cert\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.028205 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:25.028184    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:25.028291 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:25.028212    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:25.028291 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:25.028225    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ghm85 for pod openshift-network-diagnostics/network-check-target-csgtz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:25.028291 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:25.028287    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85 podName:3f818ca3-1324-41d6-9d78-46c09e9719a3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:25.528271143 +0000 UTC m=+3.128368776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ghm85" (UniqueName: "kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85") pod "network-check-target-csgtz" (UID: "3f818ca3-1324-41d6-9d78-46c09e9719a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:25.030263 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.030238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wznpm\" (UniqueName: \"kubernetes.io/projected/9b8f2f57-9508-463c-b76b-14b6207309fb-kube-api-access-wznpm\") pod \"iptables-alerter-kn6cq\" (UID: \"9b8f2f57-9508-463c-b76b-14b6207309fb\") " pod="openshift-network-operator/iptables-alerter-kn6cq"
Sep 08 13:16:25.030586 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.030566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbs99\" (UniqueName: \"kubernetes.io/projected/13d85014-a3c5-4627-80f6-62de2e79b1cf-kube-api-access-kbs99\") pod \"multus-additional-cni-plugins-95zgl\" (UID: \"13d85014-a3c5-4627-80f6-62de2e79b1cf\") " pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:25.030844 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.030823    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xjm9\" (UniqueName: \"kubernetes.io/projected/d609c6d5-329d-4100-951b-96638309f6cc-kube-api-access-9xjm9\") pod \"ovnkube-node-ms6p6\" (UID: \"d609c6d5-329d-4100-951b-96638309f6cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.031336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.031313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp9pm\" (UniqueName: \"kubernetes.io/projected/75a311a2-7bf2-4973-8db8-a4c47baf06e7-kube-api-access-zp9pm\") pod \"tuned-kzlgx\" (UID: \"75a311a2-7bf2-4973-8db8-a4c47baf06e7\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.077067 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:25.077044    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868db1574818d2dd017de1d6b08036fa.slice/crio-8439b1657609ba034c1cfbc9482364f16bc1ecf3e4ab213bcb9b150ed9f8d666 WatchSource:0}: Error finding container 8439b1657609ba034c1cfbc9482364f16bc1ecf3e4ab213bcb9b150ed9f8d666: Status 404 returned error can't find the container with id 8439b1657609ba034c1cfbc9482364f16bc1ecf3e4ab213bcb9b150ed9f8d666
Sep 08 13:16:25.124218 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.124199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2jfd4"
Sep 08 13:16:25.132407 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:25.132386    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a807ed_4b45_45f2_a302_123e9b392de3.slice/crio-35c8126e315e0be11b60d83759a17bb707c49bc2be904ea64c2f4cce50e1ca29 WatchSource:0}: Error finding container 35c8126e315e0be11b60d83759a17bb707c49bc2be904ea64c2f4cce50e1ca29: Status 404 returned error can't find the container with id 35c8126e315e0be11b60d83759a17bb707c49bc2be904ea64c2f4cce50e1ca29
Sep 08 13:16:25.147902 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.147884    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j9nqf"
Sep 08 13:16:25.153605 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.153586    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kp8g5"
Sep 08 13:16:25.155107 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:25.155089    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba25c2e_e201_4764_adb2_e8faa0761d29.slice/crio-da540afe0059cfa291d24fe3bf235a00bf6005f40b2e571a80eb52b5a4b646a5 WatchSource:0}: Error finding container da540afe0059cfa291d24fe3bf235a00bf6005f40b2e571a80eb52b5a4b646a5: Status 404 returned error can't find the container with id da540afe0059cfa291d24fe3bf235a00bf6005f40b2e571a80eb52b5a4b646a5
Sep 08 13:16:25.161999 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:25.161982    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f546ae_e7b5_4459_a35a_c16a0e7807b9.slice/crio-5bbb508cfc29a063748a42fb3c59c9bf99e8a905a5e83215617f8d9341f363b7 WatchSource:0}: Error finding container 5bbb508cfc29a063748a42fb3c59c9bf99e8a905a5e83215617f8d9341f363b7: Status 404 returned error can't find the container with id 5bbb508cfc29a063748a42fb3c59c9bf99e8a905a5e83215617f8d9341f363b7
Sep 08 13:16:25.169395 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.169378    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8"
Sep 08 13:16:25.176548 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:25.176529    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod878d9839_5817_4ecc_b9ca_0abb3ae7f1a2.slice/crio-ec6db903bebf7b906e0393b1200e652928c31841b0aeb5bff40fa15c6eef21a6 WatchSource:0}: Error finding container ec6db903bebf7b906e0393b1200e652928c31841b0aeb5bff40fa15c6eef21a6: Status 404 returned error can't find the container with id ec6db903bebf7b906e0393b1200e652928c31841b0aeb5bff40fa15c6eef21a6
Sep 08 13:16:25.184670 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.184654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:25.192371 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:25.192351    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd609c6d5_329d_4100_951b_96638309f6cc.slice/crio-70a82c110b7b5ec449127b9e67cbca7d4b286ebdbd7d988a57153abf183a99dc WatchSource:0}: Error finding container 70a82c110b7b5ec449127b9e67cbca7d4b286ebdbd7d988a57153abf183a99dc: Status 404 returned error can't find the container with id 70a82c110b7b5ec449127b9e67cbca7d4b286ebdbd7d988a57153abf183a99dc
Sep 08 13:16:25.197500 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.197481    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p6sbc"
Sep 08 13:16:25.204888 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:25.204864    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e40e15f_a782_46d6_b275_7722e86f4652.slice/crio-524a73184525fad00b6ee330275e4c5a368dad1d9f6681b1138684e8931aee79 WatchSource:0}: Error finding container 524a73184525fad00b6ee330275e4c5a368dad1d9f6681b1138684e8931aee79: Status 404 returned error can't find the container with id 524a73184525fad00b6ee330275e4c5a368dad1d9f6681b1138684e8931aee79
Sep 08 13:16:25.211854 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.211835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kzlgx"
Sep 08 13:16:25.219966 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:25.219947    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a311a2_7bf2_4973_8db8_a4c47baf06e7.slice/crio-a5c411ecb6cbc5e5b01de60f19dedbb0db4731e24f84908f48c0e588d24d1f1c WatchSource:0}: Error finding container a5c411ecb6cbc5e5b01de60f19dedbb0db4731e24f84908f48c0e588d24d1f1c: Status 404 returned error can't find the container with id a5c411ecb6cbc5e5b01de60f19dedbb0db4731e24f84908f48c0e588d24d1f1c
Sep 08 13:16:25.224089 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.224074    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:25.234756 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.234739    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-95zgl"
Sep 08 13:16:25.240579 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.240559    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kn6cq"
Sep 08 13:16:25.243250 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:25.243227    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d85014_a3c5_4627_80f6_62de2e79b1cf.slice/crio-26bcbf962818644f2bc1c300fe1f49b841182c033abf3fcd2b6e297ba8c78d7f WatchSource:0}: Error finding container 26bcbf962818644f2bc1c300fe1f49b841182c033abf3fcd2b6e297ba8c78d7f: Status 404 returned error can't find the container with id 26bcbf962818644f2bc1c300fe1f49b841182c033abf3fcd2b6e297ba8c78d7f
Sep 08 13:16:25.249260 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:25.249242    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8f2f57_9508_463c_b76b_14b6207309fb.slice/crio-58634e04ff63f964aa4588ea146d58bc07f8d29a230c1d5b50c8630130d92a19 WatchSource:0}: Error finding container 58634e04ff63f964aa4588ea146d58bc07f8d29a230c1d5b50c8630130d92a19: Status 404 returned error can't find the container with id 58634e04ff63f964aa4588ea146d58bc07f8d29a230c1d5b50c8630130d92a19
Sep 08 13:16:25.424273 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.424195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs\") pod \"network-metrics-daemon-h2td8\" (UID: \"7eeefbda-9e69-4275-a756-6effb7738d90\") " pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:25.424428 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:25.424322    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:25.424428 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:25.424371    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs podName:7eeefbda-9e69-4275-a756-6effb7738d90 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:26.424358702 +0000 UTC m=+4.024456319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs") pod "network-metrics-daemon-h2td8" (UID: "7eeefbda-9e69-4275-a756-6effb7738d90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:25.625441 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.625406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghm85\" (UniqueName: \"kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85\") pod \"network-check-target-csgtz\" (UID: \"3f818ca3-1324-41d6-9d78-46c09e9719a3\") " pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:25.625612 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:25.625599    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:25.625693 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:25.625636    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:25.625693 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:25.625652    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ghm85 for pod openshift-network-diagnostics/network-check-target-csgtz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:25.625787 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:25.625709    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85 podName:3f818ca3-1324-41d6-9d78-46c09e9719a3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:26.625690557 +0000 UTC m=+4.225788165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghm85" (UniqueName: "kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85") pod "network-check-target-csgtz" (UID: "3f818ca3-1324-41d6-9d78-46c09e9719a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:25.854844 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.854713    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:11:24 +0000 UTC" deadline="2028-04-15 18:46:18.013854284 +0000 UTC"
Sep 08 13:16:25.854844 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:25.854781    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14045h29m52.15907784s"
Sep 08 13:16:26.007751 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:26.007717    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:26.007911 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:26.007865    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:26.020650 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:26.020604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal" event={"ID":"868db1574818d2dd017de1d6b08036fa","Type":"ContainerStarted","Data":"8439b1657609ba034c1cfbc9482364f16bc1ecf3e4ab213bcb9b150ed9f8d666"}
Sep 08 13:16:26.025209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:26.024942    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kn6cq" event={"ID":"9b8f2f57-9508-463c-b76b-14b6207309fb","Type":"ContainerStarted","Data":"58634e04ff63f964aa4588ea146d58bc07f8d29a230c1d5b50c8630130d92a19"}
Sep 08 13:16:26.034355 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:26.034326    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p6sbc" event={"ID":"4e40e15f-a782-46d6-b275-7722e86f4652","Type":"ContainerStarted","Data":"524a73184525fad00b6ee330275e4c5a368dad1d9f6681b1138684e8931aee79"}
Sep 08 13:16:26.037523 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:26.037428    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6" event={"ID":"d609c6d5-329d-4100-951b-96638309f6cc","Type":"ContainerStarted","Data":"70a82c110b7b5ec449127b9e67cbca7d4b286ebdbd7d988a57153abf183a99dc"}
Sep 08 13:16:26.041230 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:26.041161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95zgl" event={"ID":"13d85014-a3c5-4627-80f6-62de2e79b1cf","Type":"ContainerStarted","Data":"26bcbf962818644f2bc1c300fe1f49b841182c033abf3fcd2b6e297ba8c78d7f"}
Sep 08 13:16:26.044369 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:26.044337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kzlgx" event={"ID":"75a311a2-7bf2-4973-8db8-a4c47baf06e7","Type":"ContainerStarted","Data":"a5c411ecb6cbc5e5b01de60f19dedbb0db4731e24f84908f48c0e588d24d1f1c"}
Sep 08 13:16:26.055010 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:26.054920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8" event={"ID":"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2","Type":"ContainerStarted","Data":"ec6db903bebf7b906e0393b1200e652928c31841b0aeb5bff40fa15c6eef21a6"}
Sep 08 13:16:26.059949 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:26.059922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kp8g5" event={"ID":"a0f546ae-e7b5-4459-a35a-c16a0e7807b9","Type":"ContainerStarted","Data":"5bbb508cfc29a063748a42fb3c59c9bf99e8a905a5e83215617f8d9341f363b7"}
Sep 08 13:16:26.064058 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:26.064005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j9nqf" event={"ID":"1ba25c2e-e201-4764-adb2-e8faa0761d29","Type":"ContainerStarted","Data":"da540afe0059cfa291d24fe3bf235a00bf6005f40b2e571a80eb52b5a4b646a5"}
Sep 08 13:16:26.067383 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:26.067361    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2jfd4" event={"ID":"d7a807ed-4b45-45f2-a302-123e9b392de3","Type":"ContainerStarted","Data":"35c8126e315e0be11b60d83759a17bb707c49bc2be904ea64c2f4cce50e1ca29"}
Sep 08 13:16:26.432741 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:26.432702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs\") pod \"network-metrics-daemon-h2td8\" (UID: \"7eeefbda-9e69-4275-a756-6effb7738d90\") " pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:26.432946 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:26.432866    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:26.432946 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:26.432931    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs podName:7eeefbda-9e69-4275-a756-6effb7738d90 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:28.432913434 +0000 UTC m=+6.033011056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs") pod "network-metrics-daemon-h2td8" (UID: "7eeefbda-9e69-4275-a756-6effb7738d90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:26.634586 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:26.634407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghm85\" (UniqueName: \"kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85\") pod \"network-check-target-csgtz\" (UID: \"3f818ca3-1324-41d6-9d78-46c09e9719a3\") " pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:26.634781 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:26.634595    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:26.634781 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:26.634614    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:26.634781 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:26.634642    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ghm85 for pod openshift-network-diagnostics/network-check-target-csgtz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:26.634781 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:26.634705    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85 podName:3f818ca3-1324-41d6-9d78-46c09e9719a3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:28.634683815 +0000 UTC m=+6.234781437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghm85" (UniqueName: "kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85") pod "network-check-target-csgtz" (UID: "3f818ca3-1324-41d6-9d78-46c09e9719a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:26.855630 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:26.855501    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:11:24 +0000 UTC" deadline="2028-03-10 07:44:26.671750494 +0000 UTC"
Sep 08 13:16:26.855630 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:26.855539    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13170h27m59.816216471s"
Sep 08 13:16:27.009349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:27.008882    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:27.009349 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:27.009011    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:28.007555 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:28.007480    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:28.008051 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:28.007668    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:28.449912 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:28.449837    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs\") pod \"network-metrics-daemon-h2td8\" (UID: \"7eeefbda-9e69-4275-a756-6effb7738d90\") " pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:28.450055 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:28.449949    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:28.450055 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:28.450018    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs podName:7eeefbda-9e69-4275-a756-6effb7738d90 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:32.449986581 +0000 UTC m=+10.050084189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs") pod "network-metrics-daemon-h2td8" (UID: "7eeefbda-9e69-4275-a756-6effb7738d90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:28.651564 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:28.651522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghm85\" (UniqueName: \"kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85\") pod \"network-check-target-csgtz\" (UID: \"3f818ca3-1324-41d6-9d78-46c09e9719a3\") " pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:28.651757 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:28.651738    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:28.651816 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:28.651761    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:28.651816 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:28.651774    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ghm85 for pod openshift-network-diagnostics/network-check-target-csgtz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:28.651918 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:28.651830    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85 podName:3f818ca3-1324-41d6-9d78-46c09e9719a3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:32.651813004 +0000 UTC m=+10.251910615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghm85" (UniqueName: "kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85") pod "network-check-target-csgtz" (UID: "3f818ca3-1324-41d6-9d78-46c09e9719a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:29.008446 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:29.008409    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:29.008919 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:29.008548    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:30.008377 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:30.008343    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:30.008546 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:30.008486    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:31.008070 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:31.008009    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:31.008289 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:31.008164    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:32.008241 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:32.008155    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:32.008844 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:32.008638    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:32.484256 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:32.484150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs\") pod \"network-metrics-daemon-h2td8\" (UID: \"7eeefbda-9e69-4275-a756-6effb7738d90\") " pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:32.484425 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:32.484326    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:32.484425 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:32.484397    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs podName:7eeefbda-9e69-4275-a756-6effb7738d90 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:40.484378134 +0000 UTC m=+18.084475757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs") pod "network-metrics-daemon-h2td8" (UID: "7eeefbda-9e69-4275-a756-6effb7738d90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:32.686073 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:32.686029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghm85\" (UniqueName: \"kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85\") pod \"network-check-target-csgtz\" (UID: \"3f818ca3-1324-41d6-9d78-46c09e9719a3\") " pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:32.686251 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:32.686240    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:32.686311 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:32.686261    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:32.686311 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:32.686274    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ghm85 for pod openshift-network-diagnostics/network-check-target-csgtz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:32.686394 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:32.686342    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85 podName:3f818ca3-1324-41d6-9d78-46c09e9719a3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:40.686325318 +0000 UTC m=+18.286422925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghm85" (UniqueName: "kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85") pod "network-check-target-csgtz" (UID: "3f818ca3-1324-41d6-9d78-46c09e9719a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:33.010085 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:33.009341    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:33.010085 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:33.009448    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:34.008302 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:34.008257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:34.008491 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:34.008389    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:35.007904 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:35.007835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:35.008242 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:35.007971    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:36.007513 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:36.007432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:36.007682 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:36.007570    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:37.008256 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:37.008224    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:37.008735 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:37.008328    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:38.008340 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:38.008300    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:38.008764 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:38.008411    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:39.007913 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:39.007879    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:39.008077 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:39.007993    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:40.008391 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:40.008357    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:40.008793 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:40.008472    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:40.540857 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:40.540811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs\") pod \"network-metrics-daemon-h2td8\" (UID: \"7eeefbda-9e69-4275-a756-6effb7738d90\") " pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:40.541078 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:40.540945    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:40.541078 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:40.541006    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs podName:7eeefbda-9e69-4275-a756-6effb7738d90 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:56.540985183 +0000 UTC m=+34.141082809 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs") pod "network-metrics-daemon-h2td8" (UID: "7eeefbda-9e69-4275-a756-6effb7738d90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:40.742866 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:40.742836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghm85\" (UniqueName: \"kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85\") pod \"network-check-target-csgtz\" (UID: \"3f818ca3-1324-41d6-9d78-46c09e9719a3\") " pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:40.743030 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:40.742987    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:40.743030 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:40.743009    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:40.743030 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:40.743019    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ghm85 for pod openshift-network-diagnostics/network-check-target-csgtz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:40.743162 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:40.743089    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85 podName:3f818ca3-1324-41d6-9d78-46c09e9719a3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:56.743067249 +0000 UTC m=+34.343164874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghm85" (UniqueName: "kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85") pod "network-check-target-csgtz" (UID: "3f818ca3-1324-41d6-9d78-46c09e9719a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:41.008238 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:41.008160    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:41.008383 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:41.008272    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:42.007917 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:42.007877    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:42.008431 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:42.008032    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:43.008680 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:43.008647    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:43.009091 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:43.008761    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:43.203291 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:43.203236    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-5gvh2"]
Sep 08 13:16:43.209208 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:43.209185    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:43.209329 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:43.209261    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gvh2" podUID="dc805ec5-4549-4f72-9f0a-eb4e03cf22bb"
Sep 08 13:16:43.261855 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:43.261785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret\") pod \"global-pull-secret-syncer-5gvh2\" (UID: \"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb\") " pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:43.261855 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:43.261820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-dbus\") pod \"global-pull-secret-syncer-5gvh2\" (UID: \"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb\") " pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:43.261855 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:43.261852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-kubelet-config\") pod \"global-pull-secret-syncer-5gvh2\" (UID: \"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb\") " pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:43.363052 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:43.362871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret\") pod \"global-pull-secret-syncer-5gvh2\" (UID: \"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb\") " pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:43.363156 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:43.363080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-dbus\") pod \"global-pull-secret-syncer-5gvh2\" (UID: \"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb\") " pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:43.363156 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:43.363128    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-kubelet-config\") pod \"global-pull-secret-syncer-5gvh2\" (UID: \"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb\") " pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:43.363252 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:43.363213    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-kubelet-config\") pod \"global-pull-secret-syncer-5gvh2\" (UID: \"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb\") " pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:43.363252 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:43.363028    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:43.363348 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:43.363290    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret podName:dc805ec5-4549-4f72-9f0a-eb4e03cf22bb nodeName:}" failed. No retries permitted until 2026-09-08 13:16:43.863270635 +0000 UTC m=+21.463368253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret") pod "global-pull-secret-syncer-5gvh2" (UID: "dc805ec5-4549-4f72-9f0a-eb4e03cf22bb") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:43.363446 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:43.363423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-dbus\") pod \"global-pull-secret-syncer-5gvh2\" (UID: \"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb\") " pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:43.867046 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:43.866888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret\") pod \"global-pull-secret-syncer-5gvh2\" (UID: \"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb\") " pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:43.867141 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:43.867055    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:43.867141 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:43.867116    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret podName:dc805ec5-4549-4f72-9f0a-eb4e03cf22bb nodeName:}" failed. No retries permitted until 2026-09-08 13:16:44.867101785 +0000 UTC m=+22.467199404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret") pod "global-pull-secret-syncer-5gvh2" (UID: "dc805ec5-4549-4f72-9f0a-eb4e03cf22bb") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:44.008235 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.008208    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:44.008373 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:44.008309    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:44.114208 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.114172    2578 generic.go:356] "Generic (PLEG): container finished" podID="868db1574818d2dd017de1d6b08036fa" containerID="ec5c51a5cb98c7e5ea2d8339cdbb67f10d6139f51fa16f71f2c5cf927c0a7cb7" exitCode=0
Sep 08 13:16:44.114751 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.114245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal" event={"ID":"868db1574818d2dd017de1d6b08036fa","Type":"ContainerDied","Data":"ec5c51a5cb98c7e5ea2d8339cdbb67f10d6139f51fa16f71f2c5cf927c0a7cb7"}
Sep 08 13:16:44.115981 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.115953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.ec2.internal" event={"ID":"bed360bd3f4f5cc25abd3af9aed5d2dd","Type":"ContainerStarted","Data":"ce255591a8eefe69a55e81f039649b30df90dbdf4600044d39b06845de0386e5"}
Sep 08 13:16:44.117719 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.117691    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p6sbc" event={"ID":"4e40e15f-a782-46d6-b275-7722e86f4652","Type":"ContainerStarted","Data":"23a120e5dae3c68f23c612cf832cb52b3dbf963411040faaef9c73e7fe738bf1"}
Sep 08 13:16:44.121273 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.121245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6" event={"ID":"d609c6d5-329d-4100-951b-96638309f6cc","Type":"ContainerStarted","Data":"77cd05afbc4e51926882ac20a6b65f90d338065c2ec59cd02a1b08319b0bf439"}
Sep 08 13:16:44.121385 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.121321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6" event={"ID":"d609c6d5-329d-4100-951b-96638309f6cc","Type":"ContainerStarted","Data":"2fceb1ef36c268847de3419f235d54a0bc892fe814991114e2e2242fffa667aa"}
Sep 08 13:16:44.121385 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.121370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6" event={"ID":"d609c6d5-329d-4100-951b-96638309f6cc","Type":"ContainerStarted","Data":"65be8631f35c89a1d603e589b2d18c2859980fe02b20b0b6e638eff7785782cd"}
Sep 08 13:16:44.121462 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.121417    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6" event={"ID":"d609c6d5-329d-4100-951b-96638309f6cc","Type":"ContainerStarted","Data":"fc572b924a748cd92f903aee18a69f17b76db7d2e0980a02b9c110b2a3fb512f"}
Sep 08 13:16:44.121503 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.121465    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6" event={"ID":"d609c6d5-329d-4100-951b-96638309f6cc","Type":"ContainerStarted","Data":"cb138c431b21b0389ad1c2657f6fa372d5f84dcedb148cdc7b8ca68d009e295b"}
Sep 08 13:16:44.121557 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.121511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6" event={"ID":"d609c6d5-329d-4100-951b-96638309f6cc","Type":"ContainerStarted","Data":"ec719173d430b4a09a45ba7b1824f60e4018888d6eb644283c5a43865f19bb58"}
Sep 08 13:16:44.122926 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.122906    2578 generic.go:356] "Generic (PLEG): container finished" podID="13d85014-a3c5-4627-80f6-62de2e79b1cf" containerID="78d1b098f6ed643160fe082d0757fdd8983cc8fc790348f74063997d0ea048bd" exitCode=0
Sep 08 13:16:44.123121 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.123097    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95zgl" event={"ID":"13d85014-a3c5-4627-80f6-62de2e79b1cf","Type":"ContainerDied","Data":"78d1b098f6ed643160fe082d0757fdd8983cc8fc790348f74063997d0ea048bd"}
Sep 08 13:16:44.125078 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.125056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kzlgx" event={"ID":"75a311a2-7bf2-4973-8db8-a4c47baf06e7","Type":"ContainerStarted","Data":"4c16a9cfe177238b74bac3f74da6d822153fb8fa3450d5481e73129f9049be3f"}
Sep 08 13:16:44.127076 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.127056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8" event={"ID":"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2","Type":"ContainerStarted","Data":"b1fc983111224df9385f3351d03aa7f9e6d2cb09259c1960f72367c7570e95a7"}
Sep 08 13:16:44.128587 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.128552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kp8g5" event={"ID":"a0f546ae-e7b5-4459-a35a-c16a0e7807b9","Type":"ContainerStarted","Data":"391fc9fef720994dffaa40730f3de50ca3aa8eb20f4173aa6f1fc17c6045b8bd"}
Sep 08 13:16:44.130160 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.130137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j9nqf" event={"ID":"1ba25c2e-e201-4764-adb2-e8faa0761d29","Type":"ContainerStarted","Data":"571fa7d2a44e5b509920e29def7d9ac74884fe2ae25728841c37ced0ca60ad3a"}
Sep 08 13:16:44.131770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.131749    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2jfd4" event={"ID":"d7a807ed-4b45-45f2-a302-123e9b392de3","Type":"ContainerStarted","Data":"8bd94d03b80ec045944c8813e964d1ee51f278f5be6dda006fa91ce09c699488"}
Sep 08 13:16:44.132340 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.132301    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-224.ec2.internal" podStartSLOduration=20.132290721 podStartE2EDuration="20.132290721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:44.132038296 +0000 UTC m=+21.732135922" watchObservedRunningTime="2026-09-08 13:16:44.132290721 +0000 UTC m=+21.732388347"
Sep 08 13:16:44.150635 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.150579    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2jfd4" podStartSLOduration=3.953983938 podStartE2EDuration="22.150569983s" totalImagesPullingTime="18.196586045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:44.150372972 +0000 UTC m=+21.750470598" watchObservedRunningTime="2026-09-08 13:16:44.150569983 +0000 UTC m=+21.750667609"
Sep 08 13:16:44.167193 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.167150    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j9nqf" podStartSLOduration=4.036761371 podStartE2EDuration="22.167133414s" totalImagesPullingTime="18.130372043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:44.166484224 +0000 UTC m=+21.766581865" watchObservedRunningTime="2026-09-08 13:16:44.167133414 +0000 UTC m=+21.767231041"
Sep 08 13:16:44.186205 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.186171    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kzlgx" podStartSLOduration=3.119858619 podStartE2EDuration="21.186158949s" totalImagesPullingTime="18.06630033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:44.185723815 +0000 UTC m=+21.785821442" watchObservedRunningTime="2026-09-08 13:16:44.186158949 +0000 UTC m=+21.786256574"
Sep 08 13:16:44.223467 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.223432    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kp8g5" podStartSLOduration=7.806154575 podStartE2EDuration="22.223419137s" totalImagesPullingTime="14.417264562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:44.223379315 +0000 UTC m=+21.823476940" watchObservedRunningTime="2026-09-08 13:16:44.223419137 +0000 UTC m=+21.823516789"
Sep 08 13:16:44.239778 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.239745    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p6sbc" podStartSLOduration=10.922624796000001 podStartE2EDuration="21.239731205s" totalImagesPullingTime="10.317106409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:44.239372705 +0000 UTC m=+21.839470333" watchObservedRunningTime="2026-09-08 13:16:44.239731205 +0000 UTC m=+21.839828843"
Sep 08 13:16:44.587476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.587329    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 13:16:44.872829 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.872641    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T13:16:44.587462384Z","UUID":"4d874a65-e966-42f5-acb9-77b519d359ae","Handler":null,"Name":"","Endpoint":""}
Sep 08 13:16:44.874402 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.874386    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 13:16:44.874488 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.874411    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 13:16:44.875035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:44.875009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret\") pod \"global-pull-secret-syncer-5gvh2\" (UID: \"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb\") " pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:44.875134 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:44.875117    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:44.875189 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:44.875179    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret podName:dc805ec5-4549-4f72-9f0a-eb4e03cf22bb nodeName:}" failed. No retries permitted until 2026-09-08 13:16:46.87516266 +0000 UTC m=+24.475260264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret") pod "global-pull-secret-syncer-5gvh2" (UID: "dc805ec5-4549-4f72-9f0a-eb4e03cf22bb") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:45.007873 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:45.007841    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:45.008038 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:45.007842    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:45.008038 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:45.007955    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:45.008166 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:45.008036    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gvh2" podUID="dc805ec5-4549-4f72-9f0a-eb4e03cf22bb"
Sep 08 13:16:45.135088 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:45.135006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kn6cq" event={"ID":"9b8f2f57-9508-463c-b76b-14b6207309fb","Type":"ContainerStarted","Data":"18e67cd348cf26dd6b1a027fdf9f460ee6692be495d67fb2b203f1be4d3e662d"}
Sep 08 13:16:45.136837 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:45.136813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8" event={"ID":"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2","Type":"ContainerStarted","Data":"26c41d768f7bdfd881d94fde269a7612bcbca6932cf3fef174d1444fea984fe9"}
Sep 08 13:16:45.156779 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:45.156726    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kn6cq" podStartSLOduration=4.127739041 podStartE2EDuration="22.156712351s" totalImagesPullingTime="18.02897331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:45.155790736 +0000 UTC m=+22.755888362" watchObservedRunningTime="2026-09-08 13:16:45.156712351 +0000 UTC m=+22.756809986"
Sep 08 13:16:45.308557 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:45.308527    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:45.418946 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:45.418869    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:46.008392 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:46.008355    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:46.008573 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:46.008489    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:46.141210 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:46.141180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal" event={"ID":"868db1574818d2dd017de1d6b08036fa","Type":"ContainerStarted","Data":"67c11a00d8686b9c5af7d5cd7babbf33c4adf22bbea0ffd2b8e3b0ca3937ab3f"}
Sep 08 13:16:46.161356 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:46.161306    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-224.ec2.internal" podStartSLOduration=22.161289058 podStartE2EDuration="22.161289058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:46.160481852 +0000 UTC m=+23.760579477" watchObservedRunningTime="2026-09-08 13:16:46.161289058 +0000 UTC m=+23.761386696"
Sep 08 13:16:46.224542 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:46.223904    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p6sbc"
Sep 08 13:16:46.226196 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:46.226121    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p6sbc"
Sep 08 13:16:46.888605 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:46.888576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret\") pod \"global-pull-secret-syncer-5gvh2\" (UID: \"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb\") " pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:46.888887 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:46.888708    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:46.888887 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:46.888768    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret podName:dc805ec5-4549-4f72-9f0a-eb4e03cf22bb nodeName:}" failed. No retries permitted until 2026-09-08 13:16:50.888747307 +0000 UTC m=+28.488844924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret") pod "global-pull-secret-syncer-5gvh2" (UID: "dc805ec5-4549-4f72-9f0a-eb4e03cf22bb") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:47.007893 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:47.007858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:47.008055 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:47.007863    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:47.008055 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:47.007979    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:47.008142 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:47.008067    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gvh2" podUID="dc805ec5-4549-4f72-9f0a-eb4e03cf22bb"
Sep 08 13:16:47.148005 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:47.147926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6" event={"ID":"d609c6d5-329d-4100-951b-96638309f6cc","Type":"ContainerStarted","Data":"4c93757d8070422c5887708509bb997084779b8608b769fd7cfadb327f9a3f67"}
Sep 08 13:16:47.150531 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:47.150503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8" event={"ID":"878d9839-5817-4ecc-b9ca-0abb3ae7f1a2","Type":"ContainerStarted","Data":"4d016622a8514af3f0e2a5abdcf65aac9a1c4e7b02c7700901f934d57f211c01"}
Sep 08 13:16:47.150758 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:47.150738    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p6sbc"
Sep 08 13:16:47.151211 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:47.151194    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p6sbc"
Sep 08 13:16:47.174378 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:47.174332    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-652z8" podStartSLOduration=7.802986572 podStartE2EDuration="24.174319765s" totalImagesPullingTime="16.371333193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:47.173873432 +0000 UTC m=+24.773971060" watchObservedRunningTime="2026-09-08 13:16:47.174319765 +0000 UTC m=+24.774417422"
Sep 08 13:16:47.254786 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:47.254753    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:47.354746 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:47.354708    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:47.712996 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:47.712952    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:47.818045 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:47.818011    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:48.008439 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:48.008371    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:48.008585 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:48.008495    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:49.007653 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:49.007606    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:49.008061 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:49.007735    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gvh2" podUID="dc805ec5-4549-4f72-9f0a-eb4e03cf22bb"
Sep 08 13:16:49.008061 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:49.007766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:49.008061 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:49.007813    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:49.373165 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:49.373095    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:49.750246 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:49.750072    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:50.007714 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:50.007642    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:50.008122 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:50.007752    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:50.160885 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:50.160849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6" event={"ID":"d609c6d5-329d-4100-951b-96638309f6cc","Type":"ContainerStarted","Data":"b78a70666e1cba5aea6298cb75fe14f343f968b02d7f9ad30909263da62ed862"}
Sep 08 13:16:50.161158 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:50.161133    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:50.161278 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:50.161204    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:50.162838 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:50.162814    2578 generic.go:356] "Generic (PLEG): container finished" podID="13d85014-a3c5-4627-80f6-62de2e79b1cf" containerID="4bfd64683326061b6027d0ccf614aa7122abf9640824b80acd80b4da094db1ee" exitCode=0
Sep 08 13:16:50.162935 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:50.162860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95zgl" event={"ID":"13d85014-a3c5-4627-80f6-62de2e79b1cf","Type":"ContainerDied","Data":"4bfd64683326061b6027d0ccf614aa7122abf9640824b80acd80b4da094db1ee"}
Sep 08 13:16:50.176304 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:50.176288    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:50.190673 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:50.190635    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6" podStartSLOduration=9.001545175 podStartE2EDuration="27.19060594s" totalImagesPullingTime="18.189060765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:50.190471789 +0000 UTC m=+27.790569414" watchObservedRunningTime="2026-09-08 13:16:50.19060594 +0000 UTC m=+27.790703560"
Sep 08 13:16:50.915845 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:50.915816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret\") pod \"global-pull-secret-syncer-5gvh2\" (UID: \"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb\") " pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:50.915991 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:50.915930    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:50.915991 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:50.915980    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret podName:dc805ec5-4549-4f72-9f0a-eb4e03cf22bb nodeName:}" failed. No retries permitted until 2026-09-08 13:16:58.915965583 +0000 UTC m=+36.516063191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret") pod "global-pull-secret-syncer-5gvh2" (UID: "dc805ec5-4549-4f72-9f0a-eb4e03cf22bb") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:51.008034 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:51.008003    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:51.008676 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:51.008048    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:51.008676 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:51.008129    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:51.008676 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:51.008233    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gvh2" podUID="dc805ec5-4549-4f72-9f0a-eb4e03cf22bb"
Sep 08 13:16:51.165160 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:51.165124    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:51.179552 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:51.179495    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:16:51.369186 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:51.369159    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h2td8"]
Sep 08 13:16:51.369308 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:51.369272    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:51.369379 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:51.369362    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:51.372797 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:51.372774    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5gvh2"]
Sep 08 13:16:51.372913 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:51.372846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:51.372955 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:51.372938    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gvh2" podUID="dc805ec5-4549-4f72-9f0a-eb4e03cf22bb"
Sep 08 13:16:51.373720 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:51.373700    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-csgtz"]
Sep 08 13:16:51.373813 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:51.373762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:51.373853 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:51.373839    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:53.011363 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:53.009875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:53.011363 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:53.010353    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:53.011363 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:53.010973    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:53.011363 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:53.011109    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gvh2" podUID="dc805ec5-4549-4f72-9f0a-eb4e03cf22bb"
Sep 08 13:16:53.011363 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:53.011208    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:53.011363 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:53.011319    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:53.171891 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:53.171854    2578 generic.go:356] "Generic (PLEG): container finished" podID="13d85014-a3c5-4627-80f6-62de2e79b1cf" containerID="9870b151208597a36e64bc9fc954b6f0402ae6305954c170d58c2734cfc3c887" exitCode=0
Sep 08 13:16:53.172029 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:53.171904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95zgl" event={"ID":"13d85014-a3c5-4627-80f6-62de2e79b1cf","Type":"ContainerDied","Data":"9870b151208597a36e64bc9fc954b6f0402ae6305954c170d58c2734cfc3c887"}
Sep 08 13:16:55.008189 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:55.008111    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:55.008614 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:55.008111    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:55.008614 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:55.008231    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gvh2" podUID="dc805ec5-4549-4f72-9f0a-eb4e03cf22bb"
Sep 08 13:16:55.008614 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:55.008110    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:55.008614 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:55.008280    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:55.008614 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:55.008377    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:55.178641 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:55.178583    2578 generic.go:356] "Generic (PLEG): container finished" podID="13d85014-a3c5-4627-80f6-62de2e79b1cf" containerID="5aff9fa7e35ce0c6e79ed79526688062890b6845d2c2351993aa5b629d25fc15" exitCode=0
Sep 08 13:16:55.178776 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:55.178659    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95zgl" event={"ID":"13d85014-a3c5-4627-80f6-62de2e79b1cf","Type":"ContainerDied","Data":"5aff9fa7e35ce0c6e79ed79526688062890b6845d2c2351993aa5b629d25fc15"}
Sep 08 13:16:56.562591 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:56.562508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs\") pod \"network-metrics-daemon-h2td8\" (UID: \"7eeefbda-9e69-4275-a756-6effb7738d90\") " pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:56.563035 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:56.562643    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:56.563035 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:56.562712    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs podName:7eeefbda-9e69-4275-a756-6effb7738d90 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:28.56269436 +0000 UTC m=+66.162791966 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs") pod "network-metrics-daemon-h2td8" (UID: "7eeefbda-9e69-4275-a756-6effb7738d90") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:56.764221 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:56.764180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghm85\" (UniqueName: \"kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85\") pod \"network-check-target-csgtz\" (UID: \"3f818ca3-1324-41d6-9d78-46c09e9719a3\") " pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:56.764389 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:56.764365    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:56.764434 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:56.764395    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:56.764434 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:56.764408    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ghm85 for pod openshift-network-diagnostics/network-check-target-csgtz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:56.764499 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:56.764477    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85 podName:3f818ca3-1324-41d6-9d78-46c09e9719a3 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:28.764457961 +0000 UTC m=+66.364555579 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghm85" (UniqueName: "kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85") pod "network-check-target-csgtz" (UID: "3f818ca3-1324-41d6-9d78-46c09e9719a3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:57.008271 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.008236    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:57.008481 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.008245    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:57.008481 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.008364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:57.008481 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:57.008360    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5gvh2" podUID="dc805ec5-4549-4f72-9f0a-eb4e03cf22bb"
Sep 08 13:16:57.008668 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:57.008477    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:16:57.008668 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:57.008573    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-csgtz" podUID="3f818ca3-1324-41d6-9d78-46c09e9719a3"
Sep 08 13:16:57.196376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.196304    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-224.ec2.internal" event="NodeReady"
Sep 08 13:16:57.196593 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.196490    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 13:16:57.243525 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.243481    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gq26f"]
Sep 08 13:16:57.261296 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.261108    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qfhsh"]
Sep 08 13:16:57.261296 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.261242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gq26f"
Sep 08 13:16:57.264286 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.264233    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 13:16:57.264286 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.264248    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 13:16:57.264533 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.264513    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8sm9m\""
Sep 08 13:16:57.274952 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.274931    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gq26f"]
Sep 08 13:16:57.275059 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.274970    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qfhsh"]
Sep 08 13:16:57.275059 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.275043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qfhsh"
Sep 08 13:16:57.277800 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.277782    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 13:16:57.278124 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.277974    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z887n\""
Sep 08 13:16:57.278124 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.277997    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 13:16:57.278841 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.278823    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 13:16:57.368123 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.368096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/175b57a5-3977-4683-addd-174bfbfbbd24-config-volume\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:16:57.368123 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.368127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:16:57.368363 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.368143    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/175b57a5-3977-4683-addd-174bfbfbbd24-tmp-dir\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:16:57.368363 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.368181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48dgh\" (UniqueName: \"kubernetes.io/projected/f0d9879c-70c8-4323-b19e-2e3c1c034566-kube-api-access-48dgh\") pod \"ingress-canary-qfhsh\" (UID: \"f0d9879c-70c8-4323-b19e-2e3c1c034566\") " pod="openshift-ingress-canary/ingress-canary-qfhsh"
Sep 08 13:16:57.368363 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.368200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsx5w\" (UniqueName: \"kubernetes.io/projected/175b57a5-3977-4683-addd-174bfbfbbd24-kube-api-access-gsx5w\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:16:57.368363 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.368248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0d9879c-70c8-4323-b19e-2e3c1c034566-cert\") pod \"ingress-canary-qfhsh\" (UID: \"f0d9879c-70c8-4323-b19e-2e3c1c034566\") " pod="openshift-ingress-canary/ingress-canary-qfhsh"
Sep 08 13:16:57.469447 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.469363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/175b57a5-3977-4683-addd-174bfbfbbd24-config-volume\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:16:57.469447 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.469404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:16:57.469447 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.469429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/175b57a5-3977-4683-addd-174bfbfbbd24-tmp-dir\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:16:57.469823 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:57.469544    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:57.469823 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.469576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48dgh\" (UniqueName: \"kubernetes.io/projected/f0d9879c-70c8-4323-b19e-2e3c1c034566-kube-api-access-48dgh\") pod \"ingress-canary-qfhsh\" (UID: \"f0d9879c-70c8-4323-b19e-2e3c1c034566\") " pod="openshift-ingress-canary/ingress-canary-qfhsh"
Sep 08 13:16:57.469823 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.469613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsx5w\" (UniqueName: \"kubernetes.io/projected/175b57a5-3977-4683-addd-174bfbfbbd24-kube-api-access-gsx5w\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:16:57.469823 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.469659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0d9879c-70c8-4323-b19e-2e3c1c034566-cert\") pod \"ingress-canary-qfhsh\" (UID: \"f0d9879c-70c8-4323-b19e-2e3c1c034566\") " pod="openshift-ingress-canary/ingress-canary-qfhsh"
Sep 08 13:16:57.469823 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:57.469679    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls podName:175b57a5-3977-4683-addd-174bfbfbbd24 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:57.969657944 +0000 UTC m=+35.569755565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls") pod "dns-default-gq26f" (UID: "175b57a5-3977-4683-addd-174bfbfbbd24") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:57.469823 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.469748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/175b57a5-3977-4683-addd-174bfbfbbd24-tmp-dir\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:16:57.470128 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.469983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/175b57a5-3977-4683-addd-174bfbfbbd24-config-volume\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:16:57.474774 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.474614    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0d9879c-70c8-4323-b19e-2e3c1c034566-cert\") pod \"ingress-canary-qfhsh\" (UID: \"f0d9879c-70c8-4323-b19e-2e3c1c034566\") " pod="openshift-ingress-canary/ingress-canary-qfhsh"
Sep 08 13:16:57.480087 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.480060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsx5w\" (UniqueName: \"kubernetes.io/projected/175b57a5-3977-4683-addd-174bfbfbbd24-kube-api-access-gsx5w\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:16:57.480210 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.480064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48dgh\" (UniqueName: \"kubernetes.io/projected/f0d9879c-70c8-4323-b19e-2e3c1c034566-kube-api-access-48dgh\") pod \"ingress-canary-qfhsh\" (UID: \"f0d9879c-70c8-4323-b19e-2e3c1c034566\") " pod="openshift-ingress-canary/ingress-canary-qfhsh"
Sep 08 13:16:57.585156 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.585127    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qfhsh"
Sep 08 13:16:57.972798 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:57.972754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:16:57.973058 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:57.972937    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:57.973058 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:57.973018    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls podName:175b57a5-3977-4683-addd-174bfbfbbd24 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:58.972996857 +0000 UTC m=+36.573094464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls") pod "dns-default-gq26f" (UID: "175b57a5-3977-4683-addd-174bfbfbbd24") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:58.980915 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:58.980873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret\") pod \"global-pull-secret-syncer-5gvh2\" (UID: \"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb\") " pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:58.981355 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:58.980932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:16:58.981355 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:58.981002    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:58.981355 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:58.981092    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret podName:dc805ec5-4549-4f72-9f0a-eb4e03cf22bb nodeName:}" failed. No retries permitted until 2026-09-08 13:17:14.981071368 +0000 UTC m=+52.581168992 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret") pod "global-pull-secret-syncer-5gvh2" (UID: "dc805ec5-4549-4f72-9f0a-eb4e03cf22bb") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:58.981355 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:58.981125    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:58.981355 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:16:58.981195    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls podName:175b57a5-3977-4683-addd-174bfbfbbd24 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:00.981175208 +0000 UTC m=+38.581272829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls") pod "dns-default-gq26f" (UID: "175b57a5-3977-4683-addd-174bfbfbbd24") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:59.008018 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:59.007952    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:16:59.008018 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:59.007965    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:16:59.008018 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:59.007959    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:16:59.011033 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:59.011013    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:16:59.012219 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:59.012179    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-486d2\""
Sep 08 13:16:59.012219 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:59.012196    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:16:59.012219 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:59.012181    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:16:59.012455 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:59.012231    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:16:59.012455 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:59.012231    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2p8w4\""
Sep 08 13:16:59.438275 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:59.438248    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qfhsh"]
Sep 08 13:16:59.442060 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:16:59.442036    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d9879c_70c8_4323_b19e_2e3c1c034566.slice/crio-723a3984740cfd531a3090956b1a0af3e8e7981af8338db3087163101b2683ff WatchSource:0}: Error finding container 723a3984740cfd531a3090956b1a0af3e8e7981af8338db3087163101b2683ff: Status 404 returned error can't find the container with id 723a3984740cfd531a3090956b1a0af3e8e7981af8338db3087163101b2683ff
Sep 08 13:16:59.893001 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:16:59.892968    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:00.192404 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:00.192165    2578 generic.go:356] "Generic (PLEG): container finished" podID="13d85014-a3c5-4627-80f6-62de2e79b1cf" containerID="3bd0b457c3b61ab48511aca67c946ed58f160f9f9db945eafd97891d3fa81496" exitCode=0
Sep 08 13:17:00.193050 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:00.192247    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95zgl" event={"ID":"13d85014-a3c5-4627-80f6-62de2e79b1cf","Type":"ContainerDied","Data":"3bd0b457c3b61ab48511aca67c946ed58f160f9f9db945eafd97891d3fa81496"}
Sep 08 13:17:00.193584 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:00.193565    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qfhsh" event={"ID":"f0d9879c-70c8-4323-b19e-2e3c1c034566","Type":"ContainerStarted","Data":"723a3984740cfd531a3090956b1a0af3e8e7981af8338db3087163101b2683ff"}
Sep 08 13:17:00.996090 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:00.996058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:17:00.996246 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:17:00.996230    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:00.996322 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:17:00.996308    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls podName:175b57a5-3977-4683-addd-174bfbfbbd24 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:04.996288652 +0000 UTC m=+42.596386276 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls") pod "dns-default-gq26f" (UID: "175b57a5-3977-4683-addd-174bfbfbbd24") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:01.199469 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:01.199436    2578 generic.go:356] "Generic (PLEG): container finished" podID="13d85014-a3c5-4627-80f6-62de2e79b1cf" containerID="1222e07dac0b4054495beb4b4b949d53ef6df2426516e9a44e4582be70c85a80" exitCode=0
Sep 08 13:17:01.199469 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:01.199477    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95zgl" event={"ID":"13d85014-a3c5-4627-80f6-62de2e79b1cf","Type":"ContainerDied","Data":"1222e07dac0b4054495beb4b4b949d53ef6df2426516e9a44e4582be70c85a80"}
Sep 08 13:17:01.319456 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:01.319425    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:01.419791 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:01.419751    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:02.204857 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:02.204803    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95zgl" event={"ID":"13d85014-a3c5-4627-80f6-62de2e79b1cf","Type":"ContainerStarted","Data":"6d6c7c6711e5e6d0cd72d40dd2098b9a245f878ec56b14a37ba4f34406c0522e"}
Sep 08 13:17:02.231036 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:02.230989    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-95zgl" podStartSLOduration=8.125793931 podStartE2EDuration="39.230975094s" totalImagesPullingTime="31.105181163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:02.230360426 +0000 UTC m=+39.830458053" watchObservedRunningTime="2026-09-08 13:17:02.230975094 +0000 UTC m=+39.831072719"
Sep 08 13:17:02.355701 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:02.355674    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:02.431635 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:02.431579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:02.708650 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:02.708606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:02.798540 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:02.798499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:02.873699 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:02.873655    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:02.974091 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:02.974001    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:03.056480 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:03.056445    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:03.706140 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:03.706098    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:03.763959 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:03.763924    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:04.505160 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:04.505119    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:04.559767 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:04.559728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:04.916071 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:04.916030    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:05.020825 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:05.020789    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:05.023753 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:05.023725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:17:05.023868 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:17:05.023856    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:05.023939 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:17:05.023922    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls podName:175b57a5-3977-4683-addd-174bfbfbbd24 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:13.023904346 +0000 UTC m=+50.624001965 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls") pod "dns-default-gq26f" (UID: "175b57a5-3977-4683-addd-174bfbfbbd24") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:06.215736 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:06.215697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qfhsh" event={"ID":"f0d9879c-70c8-4323-b19e-2e3c1c034566","Type":"ContainerStarted","Data":"39dff1e9bf091392973a2924c81badd531a566da974451955713f1a5e81972b0"}
Sep 08 13:17:06.233923 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:06.233876    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qfhsh" podStartSLOduration=2.811656057 podStartE2EDuration="9.233864077s" totalImagesPullingTime="6.42220802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:06.232638637 +0000 UTC m=+43.832736255" watchObservedRunningTime="2026-09-08 13:17:06.233864077 +0000 UTC m=+43.833961702"
Sep 08 13:17:09.102636 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:09.102594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:09.203227 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:09.203192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:13.081097 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:13.081063    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:17:13.081502 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:17:13.081180    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:13.081502 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:17:13.081231    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls podName:175b57a5-3977-4683-addd-174bfbfbbd24 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:29.081217123 +0000 UTC m=+66.681314739 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls") pod "dns-default-gq26f" (UID: "175b57a5-3977-4683-addd-174bfbfbbd24") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:14.994636 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:14.994582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret\") pod \"global-pull-secret-syncer-5gvh2\" (UID: \"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb\") " pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:17:14.997598 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:14.997569    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc805ec5-4549-4f72-9f0a-eb4e03cf22bb-original-pull-secret\") pod \"global-pull-secret-syncer-5gvh2\" (UID: \"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb\") " pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:17:15.234632 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:15.234582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5gvh2"
Sep 08 13:17:15.500745 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:15.500717    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5gvh2"]
Sep 08 13:17:15.503797 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:17:15.503766    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc805ec5_4549_4f72_9f0a_eb4e03cf22bb.slice/crio-593d78df8a90379cc966a50459e14a95cb402eedf2215fa702f2306c1675117c WatchSource:0}: Error finding container 593d78df8a90379cc966a50459e14a95cb402eedf2215fa702f2306c1675117c: Status 404 returned error can't find the container with id 593d78df8a90379cc966a50459e14a95cb402eedf2215fa702f2306c1675117c
Sep 08 13:17:16.240020 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:16.239981    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5gvh2" event={"ID":"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb","Type":"ContainerStarted","Data":"593d78df8a90379cc966a50459e14a95cb402eedf2215fa702f2306c1675117c"}
Sep 08 13:17:20.252033 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:20.251995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5gvh2" event={"ID":"dc805ec5-4549-4f72-9f0a-eb4e03cf22bb","Type":"ContainerStarted","Data":"30cd63f75367c9f5c7b3bfedce30c290519b5296e4eb7425bcee068b07dfe648"}
Sep 08 13:17:20.271209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:20.271154    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5gvh2" podStartSLOduration=33.440817383 podStartE2EDuration="37.271138694s" totalImagesPullingTime="3.830321311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:20.27038106 +0000 UTC m=+57.870478688" watchObservedRunningTime="2026-09-08 13:17:20.271138694 +0000 UTC m=+57.871236372"
Sep 08 13:17:23.182543 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:23.182510    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ms6p6"
Sep 08 13:17:23.340631 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:23.340592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:23.564850 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:23.564763    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:28.583916 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:28.583879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs\") pod \"network-metrics-daemon-h2td8\" (UID: \"7eeefbda-9e69-4275-a756-6effb7738d90\") " pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:17:28.586822 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:28.586802    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:17:28.594199 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:17:28.594181    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:17:28.594278 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:17:28.594232    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs podName:7eeefbda-9e69-4275-a756-6effb7738d90 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:32.594218313 +0000 UTC m=+130.194315931 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs") pod "network-metrics-daemon-h2td8" (UID: "7eeefbda-9e69-4275-a756-6effb7738d90") : secret "metrics-daemon-secret" not found
Sep 08 13:17:28.784876 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:28.784840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghm85\" (UniqueName: \"kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85\") pod \"network-check-target-csgtz\" (UID: \"3f818ca3-1324-41d6-9d78-46c09e9719a3\") " pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:17:28.787853 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:28.787836    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:17:28.797270 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:28.797254    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:17:28.807689 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:28.807663    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghm85\" (UniqueName: \"kubernetes.io/projected/3f818ca3-1324-41d6-9d78-46c09e9719a3-kube-api-access-ghm85\") pod \"network-check-target-csgtz\" (UID: \"3f818ca3-1324-41d6-9d78-46c09e9719a3\") " pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:17:29.023970 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:29.023944    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-486d2\""
Sep 08 13:17:29.032229 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:29.032200    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:17:29.086720 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:29.086685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:17:29.086828 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:17:29.086816    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:29.086883 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:17:29.086873    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls podName:175b57a5-3977-4683-addd-174bfbfbbd24 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:01.086858329 +0000 UTC m=+98.686955934 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls") pod "dns-default-gq26f" (UID: "175b57a5-3977-4683-addd-174bfbfbbd24") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:29.301771 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:29.301690    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-csgtz"]
Sep 08 13:17:29.305410 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:17:29.305383    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f818ca3_1324_41d6_9d78_46c09e9719a3.slice/crio-3014ec9fd9a80afd8c01b1e79f2a43f9becd3b4766bcae21de65fd72ec6df9a8 WatchSource:0}: Error finding container 3014ec9fd9a80afd8c01b1e79f2a43f9becd3b4766bcae21de65fd72ec6df9a8: Status 404 returned error can't find the container with id 3014ec9fd9a80afd8c01b1e79f2a43f9becd3b4766bcae21de65fd72ec6df9a8
Sep 08 13:17:30.279403 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:30.279368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-csgtz" event={"ID":"3f818ca3-1324-41d6-9d78-46c09e9719a3","Type":"ContainerStarted","Data":"3014ec9fd9a80afd8c01b1e79f2a43f9becd3b4766bcae21de65fd72ec6df9a8"}
Sep 08 13:17:33.288063 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:33.287960    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-csgtz" event={"ID":"3f818ca3-1324-41d6-9d78-46c09e9719a3","Type":"ContainerStarted","Data":"cf57f01f2cb756d6d158684ef929fe1e12b9ac8ce99cff3c509d3c9323c428b0"}
Sep 08 13:17:33.288433 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:33.288096    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:17:33.305207 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:17:33.305159    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-csgtz" podStartSLOduration=67.300291222 podStartE2EDuration="1m10.305146961s" totalImagesPullingTime="3.004855739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:33.304538281 +0000 UTC m=+70.904635907" watchObservedRunningTime="2026-09-08 13:17:33.305146961 +0000 UTC m=+70.905244586"
Sep 08 13:18:01.111115 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:18:01.111077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:18:01.111649 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:18:01.111271    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:18:01.111649 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:18:01.111337    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls podName:175b57a5-3977-4683-addd-174bfbfbbd24 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:05.11132039 +0000 UTC m=+162.711418012 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls") pod "dns-default-gq26f" (UID: "175b57a5-3977-4683-addd-174bfbfbbd24") : secret "dns-default-metrics-tls" not found
Sep 08 13:18:04.295520 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:18:04.295489    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-csgtz"
Sep 08 13:18:32.611437 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:18:32.611398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs\") pod \"network-metrics-daemon-h2td8\" (UID: \"7eeefbda-9e69-4275-a756-6effb7738d90\") " pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:18:32.611974 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:18:32.611554    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:18:32.611974 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:18:32.611644    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs podName:7eeefbda-9e69-4275-a756-6effb7738d90 nodeName:}" failed. No retries permitted until 2026-09-08 13:20:34.611613228 +0000 UTC m=+252.211710832 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs") pod "network-metrics-daemon-h2td8" (UID: "7eeefbda-9e69-4275-a756-6effb7738d90") : secret "metrics-daemon-secret" not found
Sep 08 13:19:00.273077 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:00.273024    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gq26f" podUID="175b57a5-3977-4683-addd-174bfbfbbd24"
Sep 08 13:19:00.490967 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:00.490937    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gq26f"
Sep 08 13:19:02.028775 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:02.028711    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-h2td8" podUID="7eeefbda-9e69-4275-a756-6effb7738d90"
Sep 08 13:19:05.121392 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:05.121354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:19:05.121803 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:05.121462    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:19:05.121803 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:05.121543    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls podName:175b57a5-3977-4683-addd-174bfbfbbd24 nodeName:}" failed. No retries permitted until 2026-09-08 13:21:07.121522372 +0000 UTC m=+284.721619994 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls") pod "dns-default-gq26f" (UID: "175b57a5-3977-4683-addd-174bfbfbbd24") : secret "dns-default-metrics-tls" not found
Sep 08 13:19:16.008278 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.008192    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:19:16.924665 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.924632    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7f8f764c4b-6nkbv"]
Sep 08 13:19:16.926306 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.926289    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-787b7b87d5-hb4px"]
Sep 08 13:19:16.926436 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.926416    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:16.928006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.927985    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:16.931340 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.931320    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 08 13:19:16.931443 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.931399    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-t8v4f\""
Sep 08 13:19:16.931955 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.931933    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 08 13:19:16.932350 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.932333    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 13:19:16.932427 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.932367    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 13:19:16.932427 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.932412    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 08 13:19:16.937322 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.937302    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 08 13:19:16.937460 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.937444    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p6qj7\""
Sep 08 13:19:16.937460 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.937453    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 13:19:16.937582 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.937551    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 13:19:16.942319 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.942298    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 13:19:16.943369 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.943346    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-787b7b87d5-hb4px"]
Sep 08 13:19:16.943660 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.943643    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 13:19:16.959739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:16.959707    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f8f764c4b-6nkbv"]
Sep 08 13:19:17.098902 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.098861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-bound-sa-token\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.098902 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.098910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:17.098902 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.098927    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-image-registry-private-configuration\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.099481 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.098991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-ca-trust-extracted\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.099481 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.099079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-certificates\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.099481 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.099121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-default-certificate\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:17.099481 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.099146    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l82c4\" (UniqueName: \"kubernetes.io/projected/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-kube-api-access-l82c4\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:17.099481 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.099174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.099481 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.099210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-trusted-ca\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.099481 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.099237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-installation-pull-secrets\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.099481 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.099273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzkkq\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-kube-api-access-gzkkq\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.099481 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.099299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:17.099481 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.099329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-stats-auth\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:17.167407 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.167377    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-8v9kw"]
Sep 08 13:19:17.168860 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.168845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.172997 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.172972    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 13:19:17.173790 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.173770    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 08 13:19:17.174036 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.174010    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 13:19:17.174253 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.174238    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 08 13:19:17.175647 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.175571    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-n84n6\""
Sep 08 13:19:17.181589 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.181570    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 08 13:19:17.196998 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.196963    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-8v9kw"]
Sep 08 13:19:17.200209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.200186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzkkq\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-kube-api-access-gzkkq\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.200310 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.200217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:17.200310 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.200239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-stats-auth\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:17.200310 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.200262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-bound-sa-token\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.200522 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:17.200399    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle podName:e9e8b582-cc4d-40fe-a2c3-62538b991ba2 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:17.700372393 +0000 UTC m=+175.300470017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle") pod "router-default-7f8f764c4b-6nkbv" (UID: "e9e8b582-cc4d-40fe-a2c3-62538b991ba2") : configmap references non-existent config key: service-ca.crt
Sep 08 13:19:17.200522 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.200470    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:17.200652 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.200532    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-image-registry-private-configuration\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.200652 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.200561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-ca-trust-extracted\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.200652 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:17.200636    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:19:17.200809 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:17.200697    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs podName:e9e8b582-cc4d-40fe-a2c3-62538b991ba2 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:17.700681717 +0000 UTC m=+175.300779345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs") pod "router-default-7f8f764c4b-6nkbv" (UID: "e9e8b582-cc4d-40fe-a2c3-62538b991ba2") : secret "router-metrics-certs-default" not found
Sep 08 13:19:17.200809 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.200751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-certificates\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.200809 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.200794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-default-certificate\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:17.200966 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.200820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l82c4\" (UniqueName: \"kubernetes.io/projected/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-kube-api-access-l82c4\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:17.200966 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.200856    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.200966 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.200891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-trusted-ca\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.200966 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.200917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-installation-pull-secrets\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.201154 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.200991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-ca-trust-extracted\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.201333 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:17.201315    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:19:17.201424 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:17.201412    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787b7b87d5-hb4px: secret "image-registry-tls" not found
Sep 08 13:19:17.201582 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:17.201564    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls podName:6d7b9eba-cdb1-4b81-86f0-2746ed4ef364 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:17.701539339 +0000 UTC m=+175.301636952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls") pod "image-registry-787b7b87d5-hb4px" (UID: "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364") : secret "image-registry-tls" not found
Sep 08 13:19:17.201710 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.201438    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-certificates\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.202919 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.202893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-trusted-ca\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.203286 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.203252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-stats-auth\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:17.203477 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.203457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-image-registry-private-configuration\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.203534 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.203485    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-installation-pull-secrets\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.203813 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.203790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-default-certificate\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:17.215058 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.215031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzkkq\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-kube-api-access-gzkkq\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.219078 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.219052    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l82c4\" (UniqueName: \"kubernetes.io/projected/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-kube-api-access-l82c4\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:17.219665 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.219645    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-bound-sa-token\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.301961 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.301919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/586b9989-b44f-4333-a970-66f5dd90f05f-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.301961 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.301967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/586b9989-b44f-4333-a970-66f5dd90f05f-serving-cert\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.302162 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.302034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgmc\" (UniqueName: \"kubernetes.io/projected/586b9989-b44f-4333-a970-66f5dd90f05f-kube-api-access-5fgmc\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.302162 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.302091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/586b9989-b44f-4333-a970-66f5dd90f05f-tmp\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.302162 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.302113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/586b9989-b44f-4333-a970-66f5dd90f05f-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.302258 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.302234    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/586b9989-b44f-4333-a970-66f5dd90f05f-snapshots\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.403257 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.403225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/586b9989-b44f-4333-a970-66f5dd90f05f-snapshots\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.403336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.403259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/586b9989-b44f-4333-a970-66f5dd90f05f-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.403336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.403284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/586b9989-b44f-4333-a970-66f5dd90f05f-serving-cert\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.403336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.403312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgmc\" (UniqueName: \"kubernetes.io/projected/586b9989-b44f-4333-a970-66f5dd90f05f-kube-api-access-5fgmc\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.404270 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.404236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/586b9989-b44f-4333-a970-66f5dd90f05f-tmp\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.404270 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.403478    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/586b9989-b44f-4333-a970-66f5dd90f05f-tmp\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.404482 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.404359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/586b9989-b44f-4333-a970-66f5dd90f05f-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.404482 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.404383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/586b9989-b44f-4333-a970-66f5dd90f05f-snapshots\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.405513 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.405180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/586b9989-b44f-4333-a970-66f5dd90f05f-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.405513 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.405458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/586b9989-b44f-4333-a970-66f5dd90f05f-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.406708 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.406674    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/586b9989-b44f-4333-a970-66f5dd90f05f-serving-cert\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.413238 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.413217    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgmc\" (UniqueName: \"kubernetes.io/projected/586b9989-b44f-4333-a970-66f5dd90f05f-kube-api-access-5fgmc\") pod \"insights-operator-76bcd6ccf5-8v9kw\" (UID: \"586b9989-b44f-4333-a970-66f5dd90f05f\") " pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.478176 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.478099    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw"
Sep 08 13:19:17.708153 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.708099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:17.708343 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.708188    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:17.708343 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.708225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:17.708343 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:17.708272    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:19:17.708343 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:17.708341    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs podName:e9e8b582-cc4d-40fe-a2c3-62538b991ba2 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:18.708325934 +0000 UTC m=+176.308423565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs") pod "router-default-7f8f764c4b-6nkbv" (UID: "e9e8b582-cc4d-40fe-a2c3-62538b991ba2") : secret "router-metrics-certs-default" not found
Sep 08 13:19:17.708483 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:17.708348    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:19:17.708483 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:17.708371    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787b7b87d5-hb4px: secret "image-registry-tls" not found
Sep 08 13:19:17.708483 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:17.708357    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle podName:e9e8b582-cc4d-40fe-a2c3-62538b991ba2 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:18.708351249 +0000 UTC m=+176.308448853 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle") pod "router-default-7f8f764c4b-6nkbv" (UID: "e9e8b582-cc4d-40fe-a2c3-62538b991ba2") : configmap references non-existent config key: service-ca.crt
Sep 08 13:19:17.708483 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:17.708438    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls podName:6d7b9eba-cdb1-4b81-86f0-2746ed4ef364 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:18.708421007 +0000 UTC m=+176.308518615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls") pod "image-registry-787b7b87d5-hb4px" (UID: "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364") : secret "image-registry-tls" not found
Sep 08 13:19:17.762912 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:17.762823    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-8v9kw"]
Sep 08 13:19:17.765668 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:19:17.765636    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586b9989_b44f_4333_a970_66f5dd90f05f.slice/crio-435f23bc74495a925cdffbf48e94e6dc36924186f28944ad2ad633b4358f2763 WatchSource:0}: Error finding container 435f23bc74495a925cdffbf48e94e6dc36924186f28944ad2ad633b4358f2763: Status 404 returned error can't find the container with id 435f23bc74495a925cdffbf48e94e6dc36924186f28944ad2ad633b4358f2763
Sep 08 13:19:18.535254 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:18.535218    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw" event={"ID":"586b9989-b44f-4333-a970-66f5dd90f05f","Type":"ContainerStarted","Data":"435f23bc74495a925cdffbf48e94e6dc36924186f28944ad2ad633b4358f2763"}
Sep 08 13:19:18.715680 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:18.715643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:18.715854 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:18.715732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:18.715854 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:18.715777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:18.715854 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:18.715816    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:19:18.715980 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:18.715884    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs podName:e9e8b582-cc4d-40fe-a2c3-62538b991ba2 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:20.715868439 +0000 UTC m=+178.315966061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs") pod "router-default-7f8f764c4b-6nkbv" (UID: "e9e8b582-cc4d-40fe-a2c3-62538b991ba2") : secret "router-metrics-certs-default" not found
Sep 08 13:19:18.715980 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:18.715894    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:19:18.715980 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:18.715918    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787b7b87d5-hb4px: secret "image-registry-tls" not found
Sep 08 13:19:18.715980 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:18.715898    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle podName:e9e8b582-cc4d-40fe-a2c3-62538b991ba2 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:20.715892772 +0000 UTC m=+178.315990376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle") pod "router-default-7f8f764c4b-6nkbv" (UID: "e9e8b582-cc4d-40fe-a2c3-62538b991ba2") : configmap references non-existent config key: service-ca.crt
Sep 08 13:19:18.716135 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:18.715983    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls podName:6d7b9eba-cdb1-4b81-86f0-2746ed4ef364 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:20.715966028 +0000 UTC m=+178.316063637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls") pod "image-registry-787b7b87d5-hb4px" (UID: "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364") : secret "image-registry-tls" not found
Sep 08 13:19:20.542057 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:20.542026    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw" event={"ID":"586b9989-b44f-4333-a970-66f5dd90f05f","Type":"ContainerStarted","Data":"9803cbda2fce6262e5b5ab247215cd012e73cb534b46d544256fe7aa6063e21d"}
Sep 08 13:19:20.563123 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:20.563079    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw" podStartSLOduration=1.599275112 podStartE2EDuration="3.563063525s" totalImagesPullingTime="1.963788413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:20.562476229 +0000 UTC m=+178.162573857" watchObservedRunningTime="2026-09-08 13:19:20.563063525 +0000 UTC m=+178.163161150"
Sep 08 13:19:20.727835 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:20.727799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:20.728014 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:20.727853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:20.728014 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:20.727885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:20.728014 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:20.727949    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:19:20.728014 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:20.727972    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787b7b87d5-hb4px: secret "image-registry-tls" not found
Sep 08 13:19:20.728014 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:20.727993    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:19:20.728191 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:20.728026    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls podName:6d7b9eba-cdb1-4b81-86f0-2746ed4ef364 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:24.728011473 +0000 UTC m=+182.328109077 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls") pod "image-registry-787b7b87d5-hb4px" (UID: "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364") : secret "image-registry-tls" not found
Sep 08 13:19:20.728191 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:20.728041    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle podName:e9e8b582-cc4d-40fe-a2c3-62538b991ba2 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:24.728035012 +0000 UTC m=+182.328132617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle") pod "router-default-7f8f764c4b-6nkbv" (UID: "e9e8b582-cc4d-40fe-a2c3-62538b991ba2") : configmap references non-existent config key: service-ca.crt
Sep 08 13:19:20.728191 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:20.728052    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs podName:e9e8b582-cc4d-40fe-a2c3-62538b991ba2 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:24.728047095 +0000 UTC m=+182.328144699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs") pod "router-default-7f8f764c4b-6nkbv" (UID: "e9e8b582-cc4d-40fe-a2c3-62538b991ba2") : secret "router-metrics-certs-default" not found
Sep 08 13:19:21.067973 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.067936    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7kmm7"]
Sep 08 13:19:21.069752 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.069735    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.074679 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.074657    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 13:19:21.074964 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.074937    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kdgsm\""
Sep 08 13:19:21.075066 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.075046    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 13:19:21.084111 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.084086    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7kmm7"]
Sep 08 13:19:21.129968 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.129943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07454356-ccf6-4e3e-8b6b-78473645fe8c-data-volume\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.130083 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.129976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqp29\" (UniqueName: \"kubernetes.io/projected/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-api-access-nqp29\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.130083 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.130002    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07454356-ccf6-4e3e-8b6b-78473645fe8c-crio-socket\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.130083 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.130042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.130195 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.130103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.230473 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.230441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.230611 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.230510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07454356-ccf6-4e3e-8b6b-78473645fe8c-data-volume\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.230611 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.230538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqp29\" (UniqueName: \"kubernetes.io/projected/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-api-access-nqp29\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.230611 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.230565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07454356-ccf6-4e3e-8b6b-78473645fe8c-crio-socket\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.230611 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.230594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.230611 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:21.230568    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:21.230837 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.230678    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/07454356-ccf6-4e3e-8b6b-78473645fe8c-crio-socket\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.230837 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:21.230689    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm podName:07454356-ccf6-4e3e-8b6b-78473645fe8c nodeName:}" failed. No retries permitted until 2026-09-08 13:19:21.730670372 +0000 UTC m=+179.330767998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7kmm7" (UID: "07454356-ccf6-4e3e-8b6b-78473645fe8c") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:21.230837 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:21.230713    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:21.230837 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:21.230765    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls podName:07454356-ccf6-4e3e-8b6b-78473645fe8c nodeName:}" failed. No retries permitted until 2026-09-08 13:19:21.730750778 +0000 UTC m=+179.330848402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kmm7" (UID: "07454356-ccf6-4e3e-8b6b-78473645fe8c") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:21.230982 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.230964    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/07454356-ccf6-4e3e-8b6b-78473645fe8c-data-volume\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.245135 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.245104    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqp29\" (UniqueName: \"kubernetes.io/projected/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-api-access-nqp29\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.733842 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.733809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.734295 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:21.733892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:21.734295 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:21.733909    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:21.734295 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:21.733971    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm podName:07454356-ccf6-4e3e-8b6b-78473645fe8c nodeName:}" failed. No retries permitted until 2026-09-08 13:19:22.733956819 +0000 UTC m=+180.334054423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7kmm7" (UID: "07454356-ccf6-4e3e-8b6b-78473645fe8c") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:21.734295 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:21.733971    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:21.734295 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:21.734013    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls podName:07454356-ccf6-4e3e-8b6b-78473645fe8c nodeName:}" failed. No retries permitted until 2026-09-08 13:19:22.734002144 +0000 UTC m=+180.334099778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kmm7" (UID: "07454356-ccf6-4e3e-8b6b-78473645fe8c") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:22.741281 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:22.741240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:22.741670 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:22.741361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:22.741670 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:22.741372    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:22.741670 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:22.741443    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm podName:07454356-ccf6-4e3e-8b6b-78473645fe8c nodeName:}" failed. No retries permitted until 2026-09-08 13:19:24.741420019 +0000 UTC m=+182.341517639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7kmm7" (UID: "07454356-ccf6-4e3e-8b6b-78473645fe8c") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:22.741670 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:22.741474    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:22.741670 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:22.741513    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls podName:07454356-ccf6-4e3e-8b6b-78473645fe8c nodeName:}" failed. No retries permitted until 2026-09-08 13:19:24.741501817 +0000 UTC m=+182.341599441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kmm7" (UID: "07454356-ccf6-4e3e-8b6b-78473645fe8c") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:23.005770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:23.005706    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j9nqf_1ba25c2e-e201-4764-adb2-e8faa0761d29/dns-node-resolver/0.log"
Sep 08 13:19:24.005761 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.005713    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kp8g5_a0f546ae-e7b5-4459-a35a-c16a0e7807b9/node-ca/0.log"
Sep 08 13:19:24.605401 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.605372    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qfhsh_f0d9879c-70c8-4323-b19e-2e3c1c034566/serve-healthcheck-canary/0.log"
Sep 08 13:19:24.668767 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.668737    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-b6km5"]
Sep 08 13:19:24.672010 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.671993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-b6km5"
Sep 08 13:19:24.675112 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.675086    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-66s22\""
Sep 08 13:19:24.675208 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.675120    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 08 13:19:24.676259 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.676238    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 08 13:19:24.676325 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.676238    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 08 13:19:24.676380 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.676336    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 08 13:19:24.680630 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.680594    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-b6km5"]
Sep 08 13:19:24.755127 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.755094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:24.755127 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.755127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80-signing-key\") pod \"service-ca-75d99c484-b6km5\" (UID: \"7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80\") " pod="openshift-service-ca/service-ca-75d99c484-b6km5"
Sep 08 13:19:24.755323 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.755146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:24.755323 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.755174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:24.755323 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.755210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:24.755323 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.755225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:24.755323 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:24.755250    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:19:24.755323 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:24.755272    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787b7b87d5-hb4px: secret "image-registry-tls" not found
Sep 08 13:19:24.755323 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.755286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80-signing-cabundle\") pod \"service-ca-75d99c484-b6km5\" (UID: \"7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80\") " pod="openshift-service-ca/service-ca-75d99c484-b6km5"
Sep 08 13:19:24.755323 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:24.755310    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:24.755645 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:24.755328    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle podName:e9e8b582-cc4d-40fe-a2c3-62538b991ba2 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:32.75530884 +0000 UTC m=+190.355406463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle") pod "router-default-7f8f764c4b-6nkbv" (UID: "e9e8b582-cc4d-40fe-a2c3-62538b991ba2") : configmap references non-existent config key: service-ca.crt
Sep 08 13:19:24.755645 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:24.755335    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:24.755645 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:24.755340    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:19:24.755645 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:24.755347    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls podName:6d7b9eba-cdb1-4b81-86f0-2746ed4ef364 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:32.75533958 +0000 UTC m=+190.355437183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls") pod "image-registry-787b7b87d5-hb4px" (UID: "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364") : secret "image-registry-tls" not found
Sep 08 13:19:24.755645 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.755404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnfnv\" (UniqueName: \"kubernetes.io/projected/7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80-kube-api-access-lnfnv\") pod \"service-ca-75d99c484-b6km5\" (UID: \"7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80\") " pod="openshift-service-ca/service-ca-75d99c484-b6km5"
Sep 08 13:19:24.755645 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:24.755452    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls podName:07454356-ccf6-4e3e-8b6b-78473645fe8c nodeName:}" failed. No retries permitted until 2026-09-08 13:19:28.755442317 +0000 UTC m=+186.355539921 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kmm7" (UID: "07454356-ccf6-4e3e-8b6b-78473645fe8c") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:24.755645 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:24.755471    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm podName:07454356-ccf6-4e3e-8b6b-78473645fe8c nodeName:}" failed. No retries permitted until 2026-09-08 13:19:28.755465969 +0000 UTC m=+186.355563573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7kmm7" (UID: "07454356-ccf6-4e3e-8b6b-78473645fe8c") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:24.755645 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:24.755480    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs podName:e9e8b582-cc4d-40fe-a2c3-62538b991ba2 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:32.755475626 +0000 UTC m=+190.355573231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs") pod "router-default-7f8f764c4b-6nkbv" (UID: "e9e8b582-cc4d-40fe-a2c3-62538b991ba2") : secret "router-metrics-certs-default" not found
Sep 08 13:19:24.855931 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.855856    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnfnv\" (UniqueName: \"kubernetes.io/projected/7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80-kube-api-access-lnfnv\") pod \"service-ca-75d99c484-b6km5\" (UID: \"7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80\") " pod="openshift-service-ca/service-ca-75d99c484-b6km5"
Sep 08 13:19:24.855931 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.855915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80-signing-key\") pod \"service-ca-75d99c484-b6km5\" (UID: \"7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80\") " pod="openshift-service-ca/service-ca-75d99c484-b6km5"
Sep 08 13:19:24.856177 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.856154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80-signing-cabundle\") pod \"service-ca-75d99c484-b6km5\" (UID: \"7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80\") " pod="openshift-service-ca/service-ca-75d99c484-b6km5"
Sep 08 13:19:24.857177 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.857158    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80-signing-cabundle\") pod \"service-ca-75d99c484-b6km5\" (UID: \"7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80\") " pod="openshift-service-ca/service-ca-75d99c484-b6km5"
Sep 08 13:19:24.858244 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.858221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80-signing-key\") pod \"service-ca-75d99c484-b6km5\" (UID: \"7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80\") " pod="openshift-service-ca/service-ca-75d99c484-b6km5"
Sep 08 13:19:24.865222 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.865200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnfnv\" (UniqueName: \"kubernetes.io/projected/7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80-kube-api-access-lnfnv\") pod \"service-ca-75d99c484-b6km5\" (UID: \"7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80\") " pod="openshift-service-ca/service-ca-75d99c484-b6km5"
Sep 08 13:19:24.981318 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:24.981280    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-b6km5"
Sep 08 13:19:25.247305 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:25.247208    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-b6km5"]
Sep 08 13:19:25.249700 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:19:25.249661    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2e0aa1_4d22_41b2_a7c7_47a8b5a54c80.slice/crio-f1a9a4be3fb7f2fbe4f17eb7fe27c0ed7cfd515478e53bb0ee63549b3836d1f5 WatchSource:0}: Error finding container f1a9a4be3fb7f2fbe4f17eb7fe27c0ed7cfd515478e53bb0ee63549b3836d1f5: Status 404 returned error can't find the container with id f1a9a4be3fb7f2fbe4f17eb7fe27c0ed7cfd515478e53bb0ee63549b3836d1f5
Sep 08 13:19:25.553632 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:25.553538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-b6km5" event={"ID":"7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80","Type":"ContainerStarted","Data":"f1a9a4be3fb7f2fbe4f17eb7fe27c0ed7cfd515478e53bb0ee63549b3836d1f5"}
Sep 08 13:19:27.561269 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:27.561185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-b6km5" event={"ID":"7d2e0aa1-4d22-41b2-a7c7-47a8b5a54c80","Type":"ContainerStarted","Data":"ea67f0ee290eef63362b2efe578b6fcb0470a982681827df7024a1bd82e5d1a9"}
Sep 08 13:19:27.579677 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:27.579615    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-b6km5" podStartSLOduration=1.519404322 podStartE2EDuration="3.579599912s" totalImagesPullingTime="2.06019559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:27.577783864 +0000 UTC m=+185.177881491" watchObservedRunningTime="2026-09-08 13:19:27.579599912 +0000 UTC m=+185.179697537"
Sep 08 13:19:28.788367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:28.788319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:28.788769 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:28.788422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:28.788769 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:28.788462    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:28.788769 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:28.788521    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:28.788769 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:28.788528    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls podName:07454356-ccf6-4e3e-8b6b-78473645fe8c nodeName:}" failed. No retries permitted until 2026-09-08 13:19:36.788511242 +0000 UTC m=+194.388608873 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kmm7" (UID: "07454356-ccf6-4e3e-8b6b-78473645fe8c") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:28.788769 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:28.788574    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm podName:07454356-ccf6-4e3e-8b6b-78473645fe8c nodeName:}" failed. No retries permitted until 2026-09-08 13:19:36.788558791 +0000 UTC m=+194.388656401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7kmm7" (UID: "07454356-ccf6-4e3e-8b6b-78473645fe8c") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:32.821023 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:32.820993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:32.821482 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:32.821031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:32.821482 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:32.821049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:32.821482 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:32.821139    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:19:32.821482 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:32.821172    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle podName:e9e8b582-cc4d-40fe-a2c3-62538b991ba2 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:48.821150822 +0000 UTC m=+206.421248443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle") pod "router-default-7f8f764c4b-6nkbv" (UID: "e9e8b582-cc4d-40fe-a2c3-62538b991ba2") : configmap references non-existent config key: service-ca.crt
Sep 08 13:19:32.821482 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:32.821209    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs podName:e9e8b582-cc4d-40fe-a2c3-62538b991ba2 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:48.821197527 +0000 UTC m=+206.421295132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs") pod "router-default-7f8f764c4b-6nkbv" (UID: "e9e8b582-cc4d-40fe-a2c3-62538b991ba2") : secret "router-metrics-certs-default" not found
Sep 08 13:19:32.823375 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:32.823351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls\") pod \"image-registry-787b7b87d5-hb4px\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") " pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:32.844953 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:32.844928    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p6qj7\""
Sep 08 13:19:32.852239 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:32.852224    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:33.127250 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:33.127181    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-787b7b87d5-hb4px"]
Sep 08 13:19:33.130916 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:19:33.130883    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7b9eba_cdb1_4b81_86f0_2746ed4ef364.slice/crio-0d6a7735c7a537cf9de928bf31f3a918627d720c74c90b104fd6313ea8086683 WatchSource:0}: Error finding container 0d6a7735c7a537cf9de928bf31f3a918627d720c74c90b104fd6313ea8086683: Status 404 returned error can't find the container with id 0d6a7735c7a537cf9de928bf31f3a918627d720c74c90b104fd6313ea8086683
Sep 08 13:19:33.579251 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:33.579215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-787b7b87d5-hb4px" event={"ID":"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364","Type":"ContainerStarted","Data":"49431d89679212ababd545b6ac22e067dd69285800366e8b5ceb8f1698659954"}
Sep 08 13:19:33.579424 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:33.579270    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-787b7b87d5-hb4px" event={"ID":"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364","Type":"ContainerStarted","Data":"0d6a7735c7a537cf9de928bf31f3a918627d720c74c90b104fd6313ea8086683"}
Sep 08 13:19:33.579424 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:33.579354    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:19:33.605043 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:33.604994    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-787b7b87d5-hb4px" podStartSLOduration=17.604980265000002 podStartE2EDuration="17.604980265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:33.603377211 +0000 UTC m=+191.203474839" watchObservedRunningTime="2026-09-08 13:19:33.604980265 +0000 UTC m=+191.205077888"
Sep 08 13:19:36.853279 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:36.853245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:36.853731 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:36.853306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:36.853731 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:36.853388    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:36.853731 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:36.853461    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls podName:07454356-ccf6-4e3e-8b6b-78473645fe8c nodeName:}" failed. No retries permitted until 2026-09-08 13:19:52.853445353 +0000 UTC m=+210.453542957 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7kmm7" (UID: "07454356-ccf6-4e3e-8b6b-78473645fe8c") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:36.853935 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:36.853911    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/07454356-ccf6-4e3e-8b6b-78473645fe8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:45.632432 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:45.632400    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-787b7b87d5-hb4px"]
Sep 08 13:19:48.845190 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:48.845138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:48.845190 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:48.845190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:48.845753 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:48.845731    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-service-ca-bundle\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:48.847485 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:48.847467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9e8b582-cc4d-40fe-a2c3-62538b991ba2-metrics-certs\") pod \"router-default-7f8f764c4b-6nkbv\" (UID: \"e9e8b582-cc4d-40fe-a2c3-62538b991ba2\") " pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:49.039110 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:49.039082    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-t8v4f\""
Sep 08 13:19:49.046563 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:49.046543    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:49.321063 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:49.321028    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f8f764c4b-6nkbv"]
Sep 08 13:19:49.323816 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:19:49.323787    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e8b582_cc4d_40fe_a2c3_62538b991ba2.slice/crio-f1395c4b8974a7a9dbe29a1e79156fb014feede0f9ff5370edb5aeae2fa5f576 WatchSource:0}: Error finding container f1395c4b8974a7a9dbe29a1e79156fb014feede0f9ff5370edb5aeae2fa5f576: Status 404 returned error can't find the container with id f1395c4b8974a7a9dbe29a1e79156fb014feede0f9ff5370edb5aeae2fa5f576
Sep 08 13:19:49.625255 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:49.625168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f8f764c4b-6nkbv" event={"ID":"e9e8b582-cc4d-40fe-a2c3-62538b991ba2","Type":"ContainerStarted","Data":"795c96a9becb908c19b6f43d30b44125ecb86a4b1adb2931129981b706dc8dc4"}
Sep 08 13:19:49.625255 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:49.625246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f8f764c4b-6nkbv" event={"ID":"e9e8b582-cc4d-40fe-a2c3-62538b991ba2","Type":"ContainerStarted","Data":"f1395c4b8974a7a9dbe29a1e79156fb014feede0f9ff5370edb5aeae2fa5f576"}
Sep 08 13:19:49.649949 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:49.649900    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7f8f764c4b-6nkbv" podStartSLOduration=33.649886119 podStartE2EDuration="33.649886119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:49.648750703 +0000 UTC m=+207.248848328" watchObservedRunningTime="2026-09-08 13:19:49.649886119 +0000 UTC m=+207.249983744"
Sep 08 13:19:50.047424 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:50.047395    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:50.049776 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:50.049751    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:50.628201 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:50.628173    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:50.629397 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:50.629372    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7f8f764c4b-6nkbv"
Sep 08 13:19:52.828571 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:52.828539    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-xl6b6"]
Sep 08 13:19:52.830267 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:52.830248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-xl6b6"
Sep 08 13:19:52.833145 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:52.833121    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 08 13:19:52.834401 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:52.834354    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2hh8p\""
Sep 08 13:19:52.841803 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:52.841780    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-xl6b6"]
Sep 08 13:19:52.874405 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:52.874380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:52.876604 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:52.876587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/07454356-ccf6-4e3e-8b6b-78473645fe8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7kmm7\" (UID: \"07454356-ccf6-4e3e-8b6b-78473645fe8c\") " pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:52.881544 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:52.881527    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kdgsm\""
Sep 08 13:19:52.888983 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:52.888969    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7kmm7"
Sep 08 13:19:52.975149 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:52.975120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4e026921-5fe3-4637-93db-76890e2b0a69-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-xl6b6\" (UID: \"4e026921-5fe3-4637-93db-76890e2b0a69\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-xl6b6"
Sep 08 13:19:53.075835 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:53.075806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4e026921-5fe3-4637-93db-76890e2b0a69-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-xl6b6\" (UID: \"4e026921-5fe3-4637-93db-76890e2b0a69\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-xl6b6"
Sep 08 13:19:53.078214 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:53.078191    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4e026921-5fe3-4637-93db-76890e2b0a69-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-xl6b6\" (UID: \"4e026921-5fe3-4637-93db-76890e2b0a69\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-xl6b6"
Sep 08 13:19:53.139147 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:53.139081    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-xl6b6"
Sep 08 13:19:53.158281 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:53.158257    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7kmm7"]
Sep 08 13:19:53.160471 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:19:53.160433    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07454356_ccf6_4e3e_8b6b_78473645fe8c.slice/crio-acbf000588488e498cf43031df9aaaccbd034bdbc5a7c1f099357f1f5ed57e38 WatchSource:0}: Error finding container acbf000588488e498cf43031df9aaaccbd034bdbc5a7c1f099357f1f5ed57e38: Status 404 returned error can't find the container with id acbf000588488e498cf43031df9aaaccbd034bdbc5a7c1f099357f1f5ed57e38
Sep 08 13:19:53.408710 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:53.408681    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-xl6b6"]
Sep 08 13:19:53.411746 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:19:53.411722    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e026921_5fe3_4637_93db_76890e2b0a69.slice/crio-fa2993774abb65406de331d6dede400692721d69a92b023a6d5f5523c8304844 WatchSource:0}: Error finding container fa2993774abb65406de331d6dede400692721d69a92b023a6d5f5523c8304844: Status 404 returned error can't find the container with id fa2993774abb65406de331d6dede400692721d69a92b023a6d5f5523c8304844
Sep 08 13:19:53.638993 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:53.638952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kmm7" event={"ID":"07454356-ccf6-4e3e-8b6b-78473645fe8c","Type":"ContainerStarted","Data":"858c2bbafd4b38738b05155cc11159eae6496b06359646ec9e2b2de10c7c4453"}
Sep 08 13:19:53.639155 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:53.639025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kmm7" event={"ID":"07454356-ccf6-4e3e-8b6b-78473645fe8c","Type":"ContainerStarted","Data":"acbf000588488e498cf43031df9aaaccbd034bdbc5a7c1f099357f1f5ed57e38"}
Sep 08 13:19:53.640085 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:53.640060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-xl6b6" event={"ID":"4e026921-5fe3-4637-93db-76890e2b0a69","Type":"ContainerStarted","Data":"fa2993774abb65406de331d6dede400692721d69a92b023a6d5f5523c8304844"}
Sep 08 13:19:55.637981 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.637903    2578 patch_prober.go:28] interesting pod/image-registry-787b7b87d5-hb4px container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 13:19:55.638318 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.637968    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-787b7b87d5-hb4px" podUID="6d7b9eba-cdb1-4b81-86f0-2746ed4ef364" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:19:55.646747 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.646718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-xl6b6" event={"ID":"4e026921-5fe3-4637-93db-76890e2b0a69","Type":"ContainerStarted","Data":"c167057b3c5fc770bd3c34224fffada9cd09394a912439356c0e593448c915c8"}
Sep 08 13:19:55.646998 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.646981    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-xl6b6"
Sep 08 13:19:55.651408 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.651390    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-xl6b6"
Sep 08 13:19:55.663441 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.663399    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-xl6b6" podStartSLOduration=1.790731975 podStartE2EDuration="3.663384514s" totalImagesPullingTime="1.872652539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:55.662804514 +0000 UTC m=+213.262902159" watchObservedRunningTime="2026-09-08 13:19:55.663384514 +0000 UTC m=+213.263482141"
Sep 08 13:19:55.934761 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.934727    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"]
Sep 08 13:19:55.936794 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.936777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"
Sep 08 13:19:55.939632 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.939598    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 08 13:19:55.939632 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.939613    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 08 13:19:55.939796 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.939695    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mrqh2\""
Sep 08 13:19:55.940754 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.940733    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 13:19:55.940876 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.940778    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 13:19:55.940876 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.940783    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 13:19:55.947045 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.947016    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"]
Sep 08 13:19:55.994688 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.994663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59279b50-1274-4310-8b9d-4bd70bf37504-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-f7k6w\" (UID: \"59279b50-1274-4310-8b9d-4bd70bf37504\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"
Sep 08 13:19:55.994785 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.994703    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvqc5\" (UniqueName: \"kubernetes.io/projected/59279b50-1274-4310-8b9d-4bd70bf37504-kube-api-access-xvqc5\") pod \"prometheus-operator-6f4946db56-f7k6w\" (UID: \"59279b50-1274-4310-8b9d-4bd70bf37504\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"
Sep 08 13:19:55.994785 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.994723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59279b50-1274-4310-8b9d-4bd70bf37504-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-f7k6w\" (UID: \"59279b50-1274-4310-8b9d-4bd70bf37504\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"
Sep 08 13:19:55.994859 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:55.994809    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/59279b50-1274-4310-8b9d-4bd70bf37504-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-f7k6w\" (UID: \"59279b50-1274-4310-8b9d-4bd70bf37504\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"
Sep 08 13:19:56.095724 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:56.095699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59279b50-1274-4310-8b9d-4bd70bf37504-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-f7k6w\" (UID: \"59279b50-1274-4310-8b9d-4bd70bf37504\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"
Sep 08 13:19:56.095834 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:56.095734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvqc5\" (UniqueName: \"kubernetes.io/projected/59279b50-1274-4310-8b9d-4bd70bf37504-kube-api-access-xvqc5\") pod \"prometheus-operator-6f4946db56-f7k6w\" (UID: \"59279b50-1274-4310-8b9d-4bd70bf37504\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"
Sep 08 13:19:56.095834 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:56.095753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59279b50-1274-4310-8b9d-4bd70bf37504-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-f7k6w\" (UID: \"59279b50-1274-4310-8b9d-4bd70bf37504\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"
Sep 08 13:19:56.095834 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:56.095789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/59279b50-1274-4310-8b9d-4bd70bf37504-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-f7k6w\" (UID: \"59279b50-1274-4310-8b9d-4bd70bf37504\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"
Sep 08 13:19:56.095971 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:56.095910    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 08 13:19:56.096020 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:19:56.095975    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59279b50-1274-4310-8b9d-4bd70bf37504-prometheus-operator-tls podName:59279b50-1274-4310-8b9d-4bd70bf37504 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:56.595955555 +0000 UTC m=+214.196053175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/59279b50-1274-4310-8b9d-4bd70bf37504-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-f7k6w" (UID: "59279b50-1274-4310-8b9d-4bd70bf37504") : secret "prometheus-operator-tls" not found
Sep 08 13:19:56.096434 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:56.096414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59279b50-1274-4310-8b9d-4bd70bf37504-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-f7k6w\" (UID: \"59279b50-1274-4310-8b9d-4bd70bf37504\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"
Sep 08 13:19:56.098278 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:56.098255    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59279b50-1274-4310-8b9d-4bd70bf37504-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-f7k6w\" (UID: \"59279b50-1274-4310-8b9d-4bd70bf37504\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"
Sep 08 13:19:56.104645 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:56.104610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvqc5\" (UniqueName: \"kubernetes.io/projected/59279b50-1274-4310-8b9d-4bd70bf37504-kube-api-access-xvqc5\") pod \"prometheus-operator-6f4946db56-f7k6w\" (UID: \"59279b50-1274-4310-8b9d-4bd70bf37504\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"
Sep 08 13:19:56.599809 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:56.599777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/59279b50-1274-4310-8b9d-4bd70bf37504-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-f7k6w\" (UID: \"59279b50-1274-4310-8b9d-4bd70bf37504\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"
Sep 08 13:19:56.602089 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:56.602071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/59279b50-1274-4310-8b9d-4bd70bf37504-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-f7k6w\" (UID: \"59279b50-1274-4310-8b9d-4bd70bf37504\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"
Sep 08 13:19:56.846817 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:56.846769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"
Sep 08 13:19:57.116321 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:57.116214    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-f7k6w"]
Sep 08 13:19:57.118669 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:19:57.118644    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59279b50_1274_4310_8b9d_4bd70bf37504.slice/crio-590af35e5e964766289946f8dcb4a97716e65b8ed0415579babc2edb0f20c911 WatchSource:0}: Error finding container 590af35e5e964766289946f8dcb4a97716e65b8ed0415579babc2edb0f20c911: Status 404 returned error can't find the container with id 590af35e5e964766289946f8dcb4a97716e65b8ed0415579babc2edb0f20c911
Sep 08 13:19:57.656218 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:57.656178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w" event={"ID":"59279b50-1274-4310-8b9d-4bd70bf37504","Type":"ContainerStarted","Data":"590af35e5e964766289946f8dcb4a97716e65b8ed0415579babc2edb0f20c911"}
Sep 08 13:19:58.660930 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:58.660891    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kmm7" event={"ID":"07454356-ccf6-4e3e-8b6b-78473645fe8c","Type":"ContainerStarted","Data":"ec70a6271537bd1b0eb23e97bd03af5204246d71d1c6ed29d7e23c874a7fd748"}
Sep 08 13:19:59.666065 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:59.666024    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7kmm7" event={"ID":"07454356-ccf6-4e3e-8b6b-78473645fe8c","Type":"ContainerStarted","Data":"4815636b902b879d241dcdd23921e73860547b53b2d9b016fbb92cf38b9daea4"}
Sep 08 13:19:59.687439 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:19:59.687394    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7kmm7" podStartSLOduration=32.995773878 podStartE2EDuration="38.687380543s" totalImagesPullingTime="5.691606665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:19:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:59.686203158 +0000 UTC m=+217.286300786" watchObservedRunningTime="2026-09-08 13:19:59.687380543 +0000 UTC m=+217.287478168"
Sep 08 13:20:03.680546 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:03.680507    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w" event={"ID":"59279b50-1274-4310-8b9d-4bd70bf37504","Type":"ContainerStarted","Data":"19d02b510798e46b8dc62472b52e40f011f3dc4b385c89c9d2aca65193a21a0e"}
Sep 08 13:20:03.681141 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:03.680589    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w" event={"ID":"59279b50-1274-4310-8b9d-4bd70bf37504","Type":"ContainerStarted","Data":"0a2c26f31aed4b8db3ddbe34451b4e0bf1fb6711d86e1ed159af7974e15d5d81"}
Sep 08 13:20:03.700984 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:03.700941    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-f7k6w" podStartSLOduration=2.575938041 podStartE2EDuration="8.700926261s" totalImagesPullingTime="6.12498822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:03.700311305 +0000 UTC m=+221.300408952" watchObservedRunningTime="2026-09-08 13:20:03.700926261 +0000 UTC m=+221.301023886"
Sep 08 13:20:05.283486 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.283452    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"]
Sep 08 13:20:05.285600 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.285579    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"
Sep 08 13:20:05.288665 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.288641    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 08 13:20:05.288772 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.288742    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 08 13:20:05.289100 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.289079    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wpw44\""
Sep 08 13:20:05.301204 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.301185    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9cv55"]
Sep 08 13:20:05.303674 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.303651    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.304231 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.304213    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"]
Sep 08 13:20:05.306340 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.306318    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"]
Sep 08 13:20:05.306445 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.306406    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vhknt\""
Sep 08 13:20:05.306445 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.306429    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.306445 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.306438    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 13:20:05.306945 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.306779    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 13:20:05.308468 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.308301    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 13:20:05.310279 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.309591    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 08 13:20:05.310279 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.309826    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 08 13:20:05.310279 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.310144    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 08 13:20:05.310945 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.310927    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-px95d\""
Sep 08 13:20:05.321903 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.321884    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"]
Sep 08 13:20:05.363541 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.363511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-node-exporter-wtmp\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.363541 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.363544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-metrics-client-ca\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.363756 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.363562    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-root\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.363756 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.363579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4584fb0f-fab8-4aae-b69f-98333a0b10ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-pg47z\" (UID: \"4584fb0f-fab8-4aae-b69f-98333a0b10ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"
Sep 08 13:20:05.363756 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.363600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m928s\" (UniqueName: \"kubernetes.io/projected/4584fb0f-fab8-4aae-b69f-98333a0b10ef-kube-api-access-m928s\") pod \"openshift-state-metrics-657fd6c54-pg47z\" (UID: \"4584fb0f-fab8-4aae-b69f-98333a0b10ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"
Sep 08 13:20:05.363756 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.363648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7msxj\" (UniqueName: \"kubernetes.io/projected/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-kube-api-access-7msxj\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.363756 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.363698    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-sys\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.364001 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.363758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.364001 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.363795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.364001 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.363825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpls8\" (UniqueName: \"kubernetes.io/projected/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-kube-api-access-xpls8\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.364001 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.363922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.364001 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.363960    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.364001 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.363989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-node-exporter-textfile\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.364332 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.364026    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.364332 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.364053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4584fb0f-fab8-4aae-b69f-98333a0b10ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pg47z\" (UID: \"4584fb0f-fab8-4aae-b69f-98333a0b10ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"
Sep 08 13:20:05.364332 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.364120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-node-exporter-tls\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.364332 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.364155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.364332 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.364186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4584fb0f-fab8-4aae-b69f-98333a0b10ef-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-pg47z\" (UID: \"4584fb0f-fab8-4aae-b69f-98333a0b10ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"
Sep 08 13:20:05.364332 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.364214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.465490 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.465458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.465490 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.465490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.465742 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.465512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpls8\" (UniqueName: \"kubernetes.io/projected/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-kube-api-access-xpls8\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.465742 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.465539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.465742 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.465676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.465958 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.465716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-node-exporter-textfile\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.465958 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.465812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.465958 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.465841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4584fb0f-fab8-4aae-b69f-98333a0b10ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pg47z\" (UID: \"4584fb0f-fab8-4aae-b69f-98333a0b10ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"
Sep 08 13:20:05.465958 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.465899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-node-exporter-tls\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.465958 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.465924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.465958 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.465950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4584fb0f-fab8-4aae-b69f-98333a0b10ef-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-pg47z\" (UID: \"4584fb0f-fab8-4aae-b69f-98333a0b10ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"
Sep 08 13:20:05.466237 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.465974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.466237 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.466026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-node-exporter-wtmp\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.466237 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.466053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-metrics-client-ca\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.466237 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.466084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-root\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.466237 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.466110    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4584fb0f-fab8-4aae-b69f-98333a0b10ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-pg47z\" (UID: \"4584fb0f-fab8-4aae-b69f-98333a0b10ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"
Sep 08 13:20:05.466237 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.466143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m928s\" (UniqueName: \"kubernetes.io/projected/4584fb0f-fab8-4aae-b69f-98333a0b10ef-kube-api-access-m928s\") pod \"openshift-state-metrics-657fd6c54-pg47z\" (UID: \"4584fb0f-fab8-4aae-b69f-98333a0b10ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"
Sep 08 13:20:05.466237 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.466175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7msxj\" (UniqueName: \"kubernetes.io/projected/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-kube-api-access-7msxj\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.466237 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.466211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-sys\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.466640 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.466304    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-sys\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.466640 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.466379    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.466640 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.466433    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-node-exporter-wtmp\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.466640 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.466441    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.466837 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.466112    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-node-exporter-textfile\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.467204 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.467022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.467204 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.467189    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-root\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.467374 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.467261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.467676 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.467652    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4584fb0f-fab8-4aae-b69f-98333a0b10ef-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-pg47z\" (UID: \"4584fb0f-fab8-4aae-b69f-98333a0b10ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"
Sep 08 13:20:05.467772 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.467739    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-metrics-client-ca\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.468783 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.468759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.468968 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.468942    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.469140 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.469109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4584fb0f-fab8-4aae-b69f-98333a0b10ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pg47z\" (UID: \"4584fb0f-fab8-4aae-b69f-98333a0b10ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"
Sep 08 13:20:05.469588 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.469567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-node-exporter-tls\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.469933 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.469912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4584fb0f-fab8-4aae-b69f-98333a0b10ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-pg47z\" (UID: \"4584fb0f-fab8-4aae-b69f-98333a0b10ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"
Sep 08 13:20:05.470217 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.470199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.474055 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.474032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpls8\" (UniqueName: \"kubernetes.io/projected/93c2041a-32e9-43bb-9d15-cf2acbcbafe4-kube-api-access-xpls8\") pod \"kube-state-metrics-74cb57df7-jj9fs\" (UID: \"93c2041a-32e9-43bb-9d15-cf2acbcbafe4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.474539 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.474517    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m928s\" (UniqueName: \"kubernetes.io/projected/4584fb0f-fab8-4aae-b69f-98333a0b10ef-kube-api-access-m928s\") pod \"openshift-state-metrics-657fd6c54-pg47z\" (UID: \"4584fb0f-fab8-4aae-b69f-98333a0b10ef\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"
Sep 08 13:20:05.475004 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.474983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7msxj\" (UniqueName: \"kubernetes.io/projected/4c3f1f57-7fe6-45a6-bd2d-e691ed68e756-kube-api-access-7msxj\") pod \"node-exporter-9cv55\" (UID: \"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756\") " pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.597076 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.596985    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"
Sep 08 13:20:05.618669 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.618645    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9cv55"
Sep 08 13:20:05.623522 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.623498    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"
Sep 08 13:20:05.629843 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:20:05.629808    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3f1f57_7fe6_45a6_bd2d_e691ed68e756.slice/crio-cac4e297b782ee2e86ae5da767b804983e567c0fec877a51c58d26ce69034aed WatchSource:0}: Error finding container cac4e297b782ee2e86ae5da767b804983e567c0fec877a51c58d26ce69034aed: Status 404 returned error can't find the container with id cac4e297b782ee2e86ae5da767b804983e567c0fec877a51c58d26ce69034aed
Sep 08 13:20:05.638823 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.638766    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:20:05.691280 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.691208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cv55" event={"ID":"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756","Type":"ContainerStarted","Data":"cac4e297b782ee2e86ae5da767b804983e567c0fec877a51c58d26ce69034aed"}
Sep 08 13:20:05.885324 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.885251    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z"]
Sep 08 13:20:05.888717 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:20:05.888691    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4584fb0f_fab8_4aae_b69f_98333a0b10ef.slice/crio-721fc0d9ded7a6224f4168b5677dd452bc9e547464c0a5a2f7de728724a3fec9 WatchSource:0}: Error finding container 721fc0d9ded7a6224f4168b5677dd452bc9e547464c0a5a2f7de728724a3fec9: Status 404 returned error can't find the container with id 721fc0d9ded7a6224f4168b5677dd452bc9e547464c0a5a2f7de728724a3fec9
Sep 08 13:20:05.914550 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:05.914530    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs"]
Sep 08 13:20:05.917666 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:20:05.917640    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c2041a_32e9_43bb_9d15_cf2acbcbafe4.slice/crio-15cf770c9346db0e6ff2da0fb08d126ae1ee205c3669c803185a79353e055a3e WatchSource:0}: Error finding container 15cf770c9346db0e6ff2da0fb08d126ae1ee205c3669c803185a79353e055a3e: Status 404 returned error can't find the container with id 15cf770c9346db0e6ff2da0fb08d126ae1ee205c3669c803185a79353e055a3e
Sep 08 13:20:06.352733 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.352696    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:20:06.355692 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.355665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.358302 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.358273    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 08 13:20:06.360501 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.360078    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 08 13:20:06.360501 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.360124    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 08 13:20:06.360501 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.360173    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 08 13:20:06.360501 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.360124    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 08 13:20:06.360501 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.360296    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 08 13:20:06.360501 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.360460    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 08 13:20:06.360885 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.360531    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 08 13:20:06.360885 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.360587    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8hstd\""
Sep 08 13:20:06.360885 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.360533    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 08 13:20:06.375870 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.375847    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:20:06.474364 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.474325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/67034717-310b-4454-b9a4-c39d7c22ed22-config-out\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.474575 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.474449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.474575 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.474543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67034717-310b-4454-b9a4-c39d7c22ed22-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.474731 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.474574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6j7g\" (UniqueName: \"kubernetes.io/projected/67034717-310b-4454-b9a4-c39d7c22ed22-kube-api-access-r6j7g\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.474731 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.474691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.474731 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.474727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67034717-310b-4454-b9a4-c39d7c22ed22-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.474913 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.474762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.474913 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.474789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-config-volume\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.474913 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.474813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.474913 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.474840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.474913 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.474867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/67034717-310b-4454-b9a4-c39d7c22ed22-tls-assets\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.474913 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.474892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/67034717-310b-4454-b9a4-c39d7c22ed22-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.475273 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.474935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-web-config\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.575925 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.575885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-config-volume\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.576108 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.575935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.576108 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.575964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.576108 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.575990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/67034717-310b-4454-b9a4-c39d7c22ed22-tls-assets\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.576108 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.576014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/67034717-310b-4454-b9a4-c39d7c22ed22-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.576323 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:20:06.576107    2578 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 08 13:20:06.576323 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.576163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-web-config\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.576323 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:20:06.576200    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-main-tls podName:67034717-310b-4454-b9a4-c39d7c22ed22 nodeName:}" failed. No retries permitted until 2026-09-08 13:20:07.076179797 +0000 UTC m=+224.676277424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "67034717-310b-4454-b9a4-c39d7c22ed22") : secret "alertmanager-main-tls" not found
Sep 08 13:20:06.576323 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.576269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/67034717-310b-4454-b9a4-c39d7c22ed22-config-out\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.576323 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.576304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.576568 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.576369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67034717-310b-4454-b9a4-c39d7c22ed22-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.576568 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.576401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6j7g\" (UniqueName: \"kubernetes.io/projected/67034717-310b-4454-b9a4-c39d7c22ed22-kube-api-access-r6j7g\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.576568 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.576414    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/67034717-310b-4454-b9a4-c39d7c22ed22-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.576568 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.576445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.576568 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.576484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67034717-310b-4454-b9a4-c39d7c22ed22-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.576568 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.576521    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.577893 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.577862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67034717-310b-4454-b9a4-c39d7c22ed22-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.578006 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.577890    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67034717-310b-4454-b9a4-c39d7c22ed22-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.579590 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.579565    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.580076 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.580034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-config-volume\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.580076 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.580051    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-web-config\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.580267 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.580249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/67034717-310b-4454-b9a4-c39d7c22ed22-config-out\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.580765 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.580744    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.581092 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.581069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/67034717-310b-4454-b9a4-c39d7c22ed22-tls-assets\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.581189 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.581071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.581247 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.581216    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.588502 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.588477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6j7g\" (UniqueName: \"kubernetes.io/projected/67034717-310b-4454-b9a4-c39d7c22ed22-kube-api-access-r6j7g\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:06.696204 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.696173    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs" event={"ID":"93c2041a-32e9-43bb-9d15-cf2acbcbafe4","Type":"ContainerStarted","Data":"15cf770c9346db0e6ff2da0fb08d126ae1ee205c3669c803185a79353e055a3e"}
Sep 08 13:20:06.698138 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.698115    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z" event={"ID":"4584fb0f-fab8-4aae-b69f-98333a0b10ef","Type":"ContainerStarted","Data":"38d09c22583c1dfa64bfd38b2d68d4fe4fc84f95d74e0202f09d58fffa7d231c"}
Sep 08 13:20:06.698247 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.698168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z" event={"ID":"4584fb0f-fab8-4aae-b69f-98333a0b10ef","Type":"ContainerStarted","Data":"1ad4a7d54d7b102b719a7765c7ce4c0e6b8a6564788880a90004c2062d81bfe7"}
Sep 08 13:20:06.698247 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:06.698188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z" event={"ID":"4584fb0f-fab8-4aae-b69f-98333a0b10ef","Type":"ContainerStarted","Data":"721fc0d9ded7a6224f4168b5677dd452bc9e547464c0a5a2f7de728724a3fec9"}
Sep 08 13:20:07.080653 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:07.080600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:07.083531 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:07.083505    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:07.268167 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:07.268133    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:20:07.705140 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:07.705077    2578 generic.go:356] "Generic (PLEG): container finished" podID="4c3f1f57-7fe6-45a6-bd2d-e691ed68e756" containerID="36cef1723c96c2e5d43fc52c2dbd1a8ad1e050774cbc3a25d8392fa0632a51d2" exitCode=0
Sep 08 13:20:07.705433 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:07.705142    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cv55" event={"ID":"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756","Type":"ContainerDied","Data":"36cef1723c96c2e5d43fc52c2dbd1a8ad1e050774cbc3a25d8392fa0632a51d2"}
Sep 08 13:20:07.707760 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:07.707733    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs" event={"ID":"93c2041a-32e9-43bb-9d15-cf2acbcbafe4","Type":"ContainerStarted","Data":"108392a2bf69ccae11faaeef7c0fc0c5337020a72d0ad4650f3c73e676caca36"}
Sep 08 13:20:07.710512 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:07.710489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z" event={"ID":"4584fb0f-fab8-4aae-b69f-98333a0b10ef","Type":"ContainerStarted","Data":"fe517e449ae8b547a8297c3a3417d59cf4a7debf02e324a54d4e554a62b17643"}
Sep 08 13:20:07.753189 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:07.753068    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pg47z" podStartSLOduration=1.166955283 podStartE2EDuration="2.753047962s" totalImagesPullingTime="1.586092679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:07.752799646 +0000 UTC m=+225.352897277" watchObservedRunningTime="2026-09-08 13:20:07.753047962 +0000 UTC m=+225.353145614"
Sep 08 13:20:07.869059 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:07.869030    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:20:07.871768 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:20:07.871742    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67034717_310b_4454_b9a4_c39d7c22ed22.slice/crio-c6ae6283e3e2e0fe18474b0b4c952020d508124357dfab41af46f4cb09551873 WatchSource:0}: Error finding container c6ae6283e3e2e0fe18474b0b4c952020d508124357dfab41af46f4cb09551873: Status 404 returned error can't find the container with id c6ae6283e3e2e0fe18474b0b4c952020d508124357dfab41af46f4cb09551873
Sep 08 13:20:08.715837 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:08.715800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"67034717-310b-4454-b9a4-c39d7c22ed22","Type":"ContainerStarted","Data":"c6ae6283e3e2e0fe18474b0b4c952020d508124357dfab41af46f4cb09551873"}
Sep 08 13:20:08.718171 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:08.718142    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cv55" event={"ID":"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756","Type":"ContainerStarted","Data":"4912d2981d622e07cf31acb77e15126b57f6d55a049f9fe610421729bf51ad47"}
Sep 08 13:20:08.718295 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:08.718187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cv55" event={"ID":"4c3f1f57-7fe6-45a6-bd2d-e691ed68e756","Type":"ContainerStarted","Data":"c2df7717056143ea84f7d5ddb17d10f6975e05042266ccde8db223bffa30c60f"}
Sep 08 13:20:08.720542 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:08.720506    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs" event={"ID":"93c2041a-32e9-43bb-9d15-cf2acbcbafe4","Type":"ContainerStarted","Data":"e97fd556eabed0cd90b6365c5131373ddefd46c59624c070d978c311e2e2db95"}
Sep 08 13:20:08.720680 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:08.720564    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs" event={"ID":"93c2041a-32e9-43bb-9d15-cf2acbcbafe4","Type":"ContainerStarted","Data":"1bf5887d4d4c8bf20cac08892eb84bc6203781c77ab1e9ce949d1f2c0c6828c3"}
Sep 08 13:20:08.749757 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:08.749710    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9cv55" podStartSLOduration=2.695183879 podStartE2EDuration="3.749693835s" totalImagesPullingTime="992.350223ms" totalInitContainerRuntime="62.159733ms" isStatefulPod=true podCreationTimestamp="2026-09-08 13:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:08.747562183 +0000 UTC m=+226.347659823" watchObservedRunningTime="2026-09-08 13:20:08.749693835 +0000 UTC m=+226.349791460"
Sep 08 13:20:08.772037 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:08.771983    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-jj9fs" podStartSLOduration=2.097268591 podStartE2EDuration="3.771964039s" totalImagesPullingTime="1.674695448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:08.771086581 +0000 UTC m=+226.371184209" watchObservedRunningTime="2026-09-08 13:20:08.771964039 +0000 UTC m=+226.372061666"
Sep 08 13:20:09.727290 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:09.727254    2578 generic.go:356] "Generic (PLEG): container finished" podID="67034717-310b-4454-b9a4-c39d7c22ed22" containerID="ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420" exitCode=0
Sep 08 13:20:09.727670 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:09.727343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"67034717-310b-4454-b9a4-c39d7c22ed22","Type":"ContainerDied","Data":"ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420"}
Sep 08 13:20:10.085743 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:10.085675    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f"]
Sep 08 13:20:10.089000 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:10.088974    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f"
Sep 08 13:20:10.092257 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:10.092239    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 08 13:20:10.092985 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:10.092962    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-66h2h\""
Sep 08 13:20:10.107642 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:10.107606    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f"]
Sep 08 13:20:10.211225 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:10.211190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9be4931c-c08f-4a6c-b84d-dcd6a9df9407-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-b2l8f\" (UID: \"9be4931c-c08f-4a6c-b84d-dcd6a9df9407\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f"
Sep 08 13:20:10.211372 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:10.211326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m82rb\" (UniqueName: \"kubernetes.io/projected/9be4931c-c08f-4a6c-b84d-dcd6a9df9407-kube-api-access-m82rb\") pod \"monitoring-plugin-85bbbbbf4-b2l8f\" (UID: \"9be4931c-c08f-4a6c-b84d-dcd6a9df9407\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f"
Sep 08 13:20:10.311954 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:10.311919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9be4931c-c08f-4a6c-b84d-dcd6a9df9407-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-b2l8f\" (UID: \"9be4931c-c08f-4a6c-b84d-dcd6a9df9407\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f"
Sep 08 13:20:10.312118 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:10.312022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m82rb\" (UniqueName: \"kubernetes.io/projected/9be4931c-c08f-4a6c-b84d-dcd6a9df9407-kube-api-access-m82rb\") pod \"monitoring-plugin-85bbbbbf4-b2l8f\" (UID: \"9be4931c-c08f-4a6c-b84d-dcd6a9df9407\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f"
Sep 08 13:20:10.312118 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:20:10.312072    2578 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 08 13:20:10.312204 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:20:10.312144    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9be4931c-c08f-4a6c-b84d-dcd6a9df9407-monitoring-plugin-cert podName:9be4931c-c08f-4a6c-b84d-dcd6a9df9407 nodeName:}" failed. No retries permitted until 2026-09-08 13:20:10.812128953 +0000 UTC m=+228.412226571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9be4931c-c08f-4a6c-b84d-dcd6a9df9407-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-b2l8f" (UID: "9be4931c-c08f-4a6c-b84d-dcd6a9df9407") : secret "monitoring-plugin-cert" not found
Sep 08 13:20:10.322871 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:10.322846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m82rb\" (UniqueName: \"kubernetes.io/projected/9be4931c-c08f-4a6c-b84d-dcd6a9df9407-kube-api-access-m82rb\") pod \"monitoring-plugin-85bbbbbf4-b2l8f\" (UID: \"9be4931c-c08f-4a6c-b84d-dcd6a9df9407\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f"
Sep 08 13:20:10.651194 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:10.651148    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-787b7b87d5-hb4px" podUID="6d7b9eba-cdb1-4b81-86f0-2746ed4ef364" containerName="registry" containerID="cri-o://49431d89679212ababd545b6ac22e067dd69285800366e8b5ceb8f1698659954" gracePeriod=30
Sep 08 13:20:10.817544 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:10.817500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9be4931c-c08f-4a6c-b84d-dcd6a9df9407-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-b2l8f\" (UID: \"9be4931c-c08f-4a6c-b84d-dcd6a9df9407\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f"
Sep 08 13:20:10.820576 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:10.820547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9be4931c-c08f-4a6c-b84d-dcd6a9df9407-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-b2l8f\" (UID: \"9be4931c-c08f-4a6c-b84d-dcd6a9df9407\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f"
Sep 08 13:20:10.906857 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:10.906795    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:20:10.997878 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:10.997848    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f"
Sep 08 13:20:11.019191 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.019158    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-certificates\") pod \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") "
Sep 08 13:20:11.019434 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.019418    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-trusted-ca\") pod \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") "
Sep 08 13:20:11.019549 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.019535    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-image-registry-private-configuration\") pod \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") "
Sep 08 13:20:11.019671 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.019657    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-ca-trust-extracted\") pod \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") "
Sep 08 13:20:11.019776 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.019762    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzkkq\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-kube-api-access-gzkkq\") pod \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") "
Sep 08 13:20:11.020333 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.019758    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364" (UID: "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:20:11.020333 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.019883    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls\") pod \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") "
Sep 08 13:20:11.020333 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.019989    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-installation-pull-secrets\") pod \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") "
Sep 08 13:20:11.020333 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.020034    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-bound-sa-token\") pod \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\" (UID: \"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364\") "
Sep 08 13:20:11.020333 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.020056    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364" (UID: "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:20:11.020333 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.020289    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-trusted-ca\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:20:11.020333 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.020308    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-certificates\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:20:11.022585 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.022557    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364" (UID: "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:20:11.022751 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.022720    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364" (UID: "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:20:11.022833 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.022809    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364" (UID: "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:20:11.022928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.022904    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364" (UID: "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:20:11.023045 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.023025    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-kube-api-access-gzkkq" (OuterVolumeSpecName: "kube-api-access-gzkkq") pod "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364" (UID: "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364"). InnerVolumeSpecName "kube-api-access-gzkkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:20:11.028742 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.028715    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364" (UID: "6d7b9eba-cdb1-4b81-86f0-2746ed4ef364"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:20:11.121118 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.120999    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-installation-pull-secrets\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:20:11.121118 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.121034    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-bound-sa-token\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:20:11.121118 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.121049    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-image-registry-private-configuration\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:20:11.121118 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.121063    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-ca-trust-extracted\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:20:11.121118 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.121077    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzkkq\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-kube-api-access-gzkkq\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:20:11.121118 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.121089    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364-registry-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:20:11.557556 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.556446    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:20:11.557556 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.556897    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d7b9eba-cdb1-4b81-86f0-2746ed4ef364" containerName="registry"
Sep 08 13:20:11.557556 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.556917    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d7b9eba-cdb1-4b81-86f0-2746ed4ef364" containerName="registry"
Sep 08 13:20:11.557556 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.556999    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d7b9eba-cdb1-4b81-86f0-2746ed4ef364" containerName="registry"
Sep 08 13:20:11.564045 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.561931    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.564648 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.564614    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 13:20:11.564802 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.564784    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 13:20:11.565067 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.565042    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-82ngavpe2jg48\""
Sep 08 13:20:11.565359 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.565332    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 13:20:11.566010 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.565992    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 13:20:11.566113 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.566047    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 13:20:11.566170 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.566145    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 13:20:11.567412 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.566552    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 13:20:11.567412 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.566662    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 13:20:11.567412 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.566770    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 13:20:11.567412 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.566790    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 13:20:11.567412 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.566813    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-88dsx\""
Sep 08 13:20:11.567412 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.566770    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 13:20:11.567412 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.567029    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 13:20:11.567412 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.567099    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 13:20:11.569744 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.569713    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 13:20:11.574372 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.574353    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 13:20:11.580356 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.580334    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:20:11.634941 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.634914    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f"]
Sep 08 13:20:11.637741 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:20:11.637718    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be4931c_c08f_4a6c_b84d_dcd6a9df9407.slice/crio-9fb9801eeab71a3f9e8f85edec56971cd91f0ebc27d769b0e6a6b4549cc85f85 WatchSource:0}: Error finding container 9fb9801eeab71a3f9e8f85edec56971cd91f0ebc27d769b0e6a6b4549cc85f85: Status 404 returned error can't find the container with id 9fb9801eeab71a3f9e8f85edec56971cd91f0ebc27d769b0e6a6b4549cc85f85
Sep 08 13:20:11.726062 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726171 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726085    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726171 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726171 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726306 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726369 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-config\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726422 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726406    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5171ade-f444-4e08-8975-2632ddf7cb72-config-out\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726516 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726516 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726600 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726600 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-web-config\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726708 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5171ade-f444-4e08-8975-2632ddf7cb72-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726708 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726853 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwb8n\" (UniqueName: \"kubernetes.io/projected/a5171ade-f444-4e08-8975-2632ddf7cb72-kube-api-access-kwb8n\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726853 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726853 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726786    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726853 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.726853 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.727187 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.726885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.740896 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.740835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"67034717-310b-4454-b9a4-c39d7c22ed22","Type":"ContainerStarted","Data":"67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade"}
Sep 08 13:20:11.741013 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.740976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"67034717-310b-4454-b9a4-c39d7c22ed22","Type":"ContainerStarted","Data":"be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1"}
Sep 08 13:20:11.741154 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.741041    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"67034717-310b-4454-b9a4-c39d7c22ed22","Type":"ContainerStarted","Data":"b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2"}
Sep 08 13:20:11.741154 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.741101    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"67034717-310b-4454-b9a4-c39d7c22ed22","Type":"ContainerStarted","Data":"9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c"}
Sep 08 13:20:11.742674 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.742646    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f" event={"ID":"9be4931c-c08f-4a6c-b84d-dcd6a9df9407","Type":"ContainerStarted","Data":"9fb9801eeab71a3f9e8f85edec56971cd91f0ebc27d769b0e6a6b4549cc85f85"}
Sep 08 13:20:11.744157 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.744126    2578 generic.go:356] "Generic (PLEG): container finished" podID="6d7b9eba-cdb1-4b81-86f0-2746ed4ef364" containerID="49431d89679212ababd545b6ac22e067dd69285800366e8b5ceb8f1698659954" exitCode=0
Sep 08 13:20:11.744261 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.744195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-787b7b87d5-hb4px" event={"ID":"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364","Type":"ContainerDied","Data":"49431d89679212ababd545b6ac22e067dd69285800366e8b5ceb8f1698659954"}
Sep 08 13:20:11.744261 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.744234    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-787b7b87d5-hb4px" event={"ID":"6d7b9eba-cdb1-4b81-86f0-2746ed4ef364","Type":"ContainerDied","Data":"0d6a7735c7a537cf9de928bf31f3a918627d720c74c90b104fd6313ea8086683"}
Sep 08 13:20:11.744261 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.744256    2578 scope.go:117] "RemoveContainer" containerID="49431d89679212ababd545b6ac22e067dd69285800366e8b5ceb8f1698659954"
Sep 08 13:20:11.744505 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.744485    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787b7b87d5-hb4px"
Sep 08 13:20:11.762514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.762496    2578 scope.go:117] "RemoveContainer" containerID="49431d89679212ababd545b6ac22e067dd69285800366e8b5ceb8f1698659954"
Sep 08 13:20:11.762936 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:20:11.762903    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49431d89679212ababd545b6ac22e067dd69285800366e8b5ceb8f1698659954\": container with ID starting with 49431d89679212ababd545b6ac22e067dd69285800366e8b5ceb8f1698659954 not found: ID does not exist" containerID="49431d89679212ababd545b6ac22e067dd69285800366e8b5ceb8f1698659954"
Sep 08 13:20:11.763038 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.763007    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49431d89679212ababd545b6ac22e067dd69285800366e8b5ceb8f1698659954"} err="failed to get container status \"49431d89679212ababd545b6ac22e067dd69285800366e8b5ceb8f1698659954\": rpc error: code = NotFound desc = could not find container \"49431d89679212ababd545b6ac22e067dd69285800366e8b5ceb8f1698659954\": container with ID starting with 49431d89679212ababd545b6ac22e067dd69285800366e8b5ceb8f1698659954 not found: ID does not exist"
Sep 08 13:20:11.775445 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.775416    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-787b7b87d5-hb4px"]
Sep 08 13:20:11.778543 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.778525    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-787b7b87d5-hb4px"]
Sep 08 13:20:11.827432 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.827400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-config\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.827432 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.827438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5171ade-f444-4e08-8975-2632ddf7cb72-config-out\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.827992 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.827465    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.827992 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.827483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.827992 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.827503    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.827992 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.827518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.827992 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.827536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-web-config\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.827992 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.827565    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5171ade-f444-4e08-8975-2632ddf7cb72-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.827992 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.827588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.828351 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.828246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.828465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.828440    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.828525 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.828502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwb8n\" (UniqueName: \"kubernetes.io/projected/a5171ade-f444-4e08-8975-2632ddf7cb72-kube-api-access-kwb8n\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.828585 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.828535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.828585 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.828561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.828720 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.828587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.828720 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.828653    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.828720 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.828653    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.828720 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.828710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.828944 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.828740    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.828944 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.828777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.828944 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.828807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.828944 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.828844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.828944 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.828887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.829594 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.829537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.830796 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.830130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.831505 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.831169    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.831978 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.831959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.832325 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.832232    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5171ade-f444-4e08-8975-2632ddf7cb72-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.833014 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.832459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-config\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.833014 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.832724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.833014 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.832960    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.833224 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.833139    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-web-config\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.833754 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.833710    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.833848 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.833784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.834491 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.834450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.834740 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.834720    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5171ade-f444-4e08-8975-2632ddf7cb72-config-out\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.835200 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.835157    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.835280 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.835243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.836644 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.836601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.838457 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.838435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwb8n\" (UniqueName: \"kubernetes.io/projected/a5171ade-f444-4e08-8975-2632ddf7cb72-kube-api-access-kwb8n\") pod \"prometheus-k8s-0\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:11.876549 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:11.876511    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:12.162408 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:12.162274    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:20:12.164792 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:20:12.164765    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5171ade_f444_4e08_8975_2632ddf7cb72.slice/crio-3cae3651026e6343bcaaf8753812cb85c89663b43c58fa1d2cc20726ac5a27f2 WatchSource:0}: Error finding container 3cae3651026e6343bcaaf8753812cb85c89663b43c58fa1d2cc20726ac5a27f2: Status 404 returned error can't find the container with id 3cae3651026e6343bcaaf8753812cb85c89663b43c58fa1d2cc20726ac5a27f2
Sep 08 13:20:12.756219 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:12.756180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"67034717-310b-4454-b9a4-c39d7c22ed22","Type":"ContainerStarted","Data":"236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb"}
Sep 08 13:20:12.756409 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:12.756289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"67034717-310b-4454-b9a4-c39d7c22ed22","Type":"ContainerStarted","Data":"e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d"}
Sep 08 13:20:12.757741 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:12.757715    2578 generic.go:356] "Generic (PLEG): container finished" podID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerID="b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694" exitCode=0
Sep 08 13:20:12.757860 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:12.757784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5171ade-f444-4e08-8975-2632ddf7cb72","Type":"ContainerDied","Data":"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694"}
Sep 08 13:20:12.757860 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:12.757835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5171ade-f444-4e08-8975-2632ddf7cb72","Type":"ContainerStarted","Data":"3cae3651026e6343bcaaf8753812cb85c89663b43c58fa1d2cc20726ac5a27f2"}
Sep 08 13:20:12.790053 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:12.790014    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.227101117 podStartE2EDuration="6.790001112s" totalImagesPullingTime="3.544006252s" totalInitContainerRuntime="18.893743ms" isStatefulPod=false podCreationTimestamp="2026-09-08 13:20:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:12.788239403 +0000 UTC m=+230.388337030" watchObservedRunningTime="2026-09-08 13:20:12.790001112 +0000 UTC m=+230.390098737"
Sep 08 13:20:13.012435 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:13.012403    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d7b9eba-cdb1-4b81-86f0-2746ed4ef364" path="/var/lib/kubelet/pods/6d7b9eba-cdb1-4b81-86f0-2746ed4ef364/volumes"
Sep 08 13:20:15.774717 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:15.774677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f" event={"ID":"9be4931c-c08f-4a6c-b84d-dcd6a9df9407","Type":"ContainerStarted","Data":"bea724d6f4853b6f58f80873695f4c561499611667d7b5657ddf7995ab758e2c"}
Sep 08 13:20:15.775191 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:15.774921    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f"
Sep 08 13:20:15.780268 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:15.780243    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f"
Sep 08 13:20:15.793859 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:15.793821    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-b2l8f" podStartSLOduration=2.53693156 podStartE2EDuration="5.793806657s" totalImagesPullingTime="3.256875097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:15.792636112 +0000 UTC m=+233.392733730" watchObservedRunningTime="2026-09-08 13:20:15.793806657 +0000 UTC m=+233.393904302"
Sep 08 13:20:17.788762 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:17.788726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5171ade-f444-4e08-8975-2632ddf7cb72","Type":"ContainerStarted","Data":"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"}
Sep 08 13:20:17.789178 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:17.788811    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5171ade-f444-4e08-8975-2632ddf7cb72","Type":"ContainerStarted","Data":"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"}
Sep 08 13:20:25.824879 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:25.824835    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5171ade-f444-4e08-8975-2632ddf7cb72","Type":"ContainerStarted","Data":"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"}
Sep 08 13:20:25.825350 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:25.824904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5171ade-f444-4e08-8975-2632ddf7cb72","Type":"ContainerStarted","Data":"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"}
Sep 08 13:20:25.825350 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:25.824929    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5171ade-f444-4e08-8975-2632ddf7cb72","Type":"ContainerStarted","Data":"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"}
Sep 08 13:20:25.825350 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:25.824952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5171ade-f444-4e08-8975-2632ddf7cb72","Type":"ContainerStarted","Data":"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"}
Sep 08 13:20:25.878417 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:25.878365    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.865609695 podStartE2EDuration="14.878348668s" totalImagesPullingTime="11.994909577s" totalInitContainerRuntime="17.829396ms" isStatefulPod=false podCreationTimestamp="2026-09-08 13:20:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:25.877145395 +0000 UTC m=+243.477243024" watchObservedRunningTime="2026-09-08 13:20:25.878348668 +0000 UTC m=+243.478446293"
Sep 08 13:20:26.876668 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:26.876594    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:20:34.627043 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:34.627000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs\") pod \"network-metrics-daemon-h2td8\" (UID: \"7eeefbda-9e69-4275-a756-6effb7738d90\") " pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:20:34.629324 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:34.629304    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eeefbda-9e69-4275-a756-6effb7738d90-metrics-certs\") pod \"network-metrics-daemon-h2td8\" (UID: \"7eeefbda-9e69-4275-a756-6effb7738d90\") " pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:20:34.914179 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:34.914151    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2p8w4\""
Sep 08 13:20:34.919772 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:34.919751    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h2td8"
Sep 08 13:20:35.193178 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:35.193101    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h2td8"]
Sep 08 13:20:35.196136 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:20:35.196112    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eeefbda_9e69_4275_a756_6effb7738d90.slice/crio-e284f7b5b7a98ed5b5f5b823a0ced111886a0aedad3a818174837550f72c936a WatchSource:0}: Error finding container e284f7b5b7a98ed5b5f5b823a0ced111886a0aedad3a818174837550f72c936a: Status 404 returned error can't find the container with id e284f7b5b7a98ed5b5f5b823a0ced111886a0aedad3a818174837550f72c936a
Sep 08 13:20:35.871451 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:35.871407    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h2td8" event={"ID":"7eeefbda-9e69-4275-a756-6effb7738d90","Type":"ContainerStarted","Data":"e284f7b5b7a98ed5b5f5b823a0ced111886a0aedad3a818174837550f72c936a"}
Sep 08 13:20:36.876678 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:36.876640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h2td8" event={"ID":"7eeefbda-9e69-4275-a756-6effb7738d90","Type":"ContainerStarted","Data":"cd4cf0b00aef1a9251c840d3df6fbe132a376505c94a8f8b073ab09e0471f356"}
Sep 08 13:20:36.877153 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:36.876691    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h2td8" event={"ID":"7eeefbda-9e69-4275-a756-6effb7738d90","Type":"ContainerStarted","Data":"40c95b1f9ac78bbe2754cc222f2ecbeb056a6be1a5a915002bdd4931aa035aa3"}
Sep 08 13:20:36.906784 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:36.906727    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h2td8" podStartSLOduration=253.973550198 podStartE2EDuration="4m14.906705441s" totalImagesPullingTime="933.155243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:36.904712606 +0000 UTC m=+254.504810235" watchObservedRunningTime="2026-09-08 13:20:36.906705441 +0000 UTC m=+254.506803084"
Sep 08 13:20:40.893018 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:40.892929    2578 generic.go:356] "Generic (PLEG): container finished" podID="586b9989-b44f-4333-a970-66f5dd90f05f" containerID="9803cbda2fce6262e5b5ab247215cd012e73cb534b46d544256fe7aa6063e21d" exitCode=0
Sep 08 13:20:40.893018 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:40.892977    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw" event={"ID":"586b9989-b44f-4333-a970-66f5dd90f05f","Type":"ContainerDied","Data":"9803cbda2fce6262e5b5ab247215cd012e73cb534b46d544256fe7aa6063e21d"}
Sep 08 13:20:40.893404 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:40.893296    2578 scope.go:117] "RemoveContainer" containerID="9803cbda2fce6262e5b5ab247215cd012e73cb534b46d544256fe7aa6063e21d"
Sep 08 13:20:41.899332 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:41.899295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-8v9kw" event={"ID":"586b9989-b44f-4333-a970-66f5dd90f05f","Type":"ContainerStarted","Data":"4673498a11f1c6768ea56594f5c96bd319d57378ca8c7ade93f90982db496ac2"}
Sep 08 13:20:41.920157 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:20:41.920131    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j9nqf_1ba25c2e-e201-4764-adb2-e8faa0761d29/dns-node-resolver/0.log"
Sep 08 13:21:03.491834 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:21:03.491779    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gq26f" podUID="175b57a5-3977-4683-addd-174bfbfbbd24"
Sep 08 13:21:03.995842 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:03.995810    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gq26f"
Sep 08 13:21:07.211589 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:07.211547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:21:07.213991 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:07.213963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/175b57a5-3977-4683-addd-174bfbfbbd24-metrics-tls\") pod \"dns-default-gq26f\" (UID: \"175b57a5-3977-4683-addd-174bfbfbbd24\") " pod="openshift-dns/dns-default-gq26f"
Sep 08 13:21:07.299010 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:07.298980    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8sm9m\""
Sep 08 13:21:07.306347 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:07.306326    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gq26f"
Sep 08 13:21:07.584199 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:07.584166    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gq26f"]
Sep 08 13:21:07.586777 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:21:07.586748    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175b57a5_3977_4683_addd_174bfbfbbd24.slice/crio-f4d6b1622b6e0cf185fc768d5a4d51efdb4da29628b0bcd6d181cf54b2d5b4d6 WatchSource:0}: Error finding container f4d6b1622b6e0cf185fc768d5a4d51efdb4da29628b0bcd6d181cf54b2d5b4d6: Status 404 returned error can't find the container with id f4d6b1622b6e0cf185fc768d5a4d51efdb4da29628b0bcd6d181cf54b2d5b4d6
Sep 08 13:21:08.011892 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:08.011857    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gq26f" event={"ID":"175b57a5-3977-4683-addd-174bfbfbbd24","Type":"ContainerStarted","Data":"f4d6b1622b6e0cf185fc768d5a4d51efdb4da29628b0bcd6d181cf54b2d5b4d6"}
Sep 08 13:21:10.022487 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:10.022446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gq26f" event={"ID":"175b57a5-3977-4683-addd-174bfbfbbd24","Type":"ContainerStarted","Data":"cd846e4fcfd3285c5b407d2246d3b5b7aa57f450ce40d3c4da9d46f82aa0564a"}
Sep 08 13:21:10.022948 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:10.022501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gq26f" event={"ID":"175b57a5-3977-4683-addd-174bfbfbbd24","Type":"ContainerStarted","Data":"894c5b3d02ad38ffb1169ed40087b506ec3d2cb3c2974abc8d9e51da20202a33"}
Sep 08 13:21:10.022948 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:10.022561    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gq26f"
Sep 08 13:21:10.041694 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:10.041613    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gq26f" podStartSLOduration=251.535953392 podStartE2EDuration="4m13.041600006s" totalImagesPullingTime="1.505646614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:21:10.040786938 +0000 UTC m=+287.640884565" watchObservedRunningTime="2026-09-08 13:21:10.041600006 +0000 UTC m=+287.641697631"
Sep 08 13:21:11.877461 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:11.877428    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:11.893048 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:11.893023    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:12.045704 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:12.045677    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:20.028493 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:20.028462    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gq26f"
Sep 08 13:21:22.842470 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:22.842435    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 13:21:25.778700 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:25.778661    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:21:25.781194 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:25.778997    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="alertmanager" containerID="cri-o://9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c" gracePeriod=120
Sep 08 13:21:25.781194 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:25.779056    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="kube-rbac-proxy-web" containerID="cri-o://be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1" gracePeriod=120
Sep 08 13:21:25.781194 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:25.779135    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="prom-label-proxy" containerID="cri-o://236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb" gracePeriod=120
Sep 08 13:21:25.781194 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:25.779058    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="kube-rbac-proxy-metric" containerID="cri-o://e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d" gracePeriod=120
Sep 08 13:21:25.781194 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:25.779069    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="kube-rbac-proxy" containerID="cri-o://67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade" gracePeriod=120
Sep 08 13:21:25.781194 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:25.779076    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="config-reloader" containerID="cri-o://b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2" gracePeriod=120
Sep 08 13:21:26.085399 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:26.085314    2578 generic.go:356] "Generic (PLEG): container finished" podID="67034717-310b-4454-b9a4-c39d7c22ed22" containerID="236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb" exitCode=0
Sep 08 13:21:26.085399 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:26.085339    2578 generic.go:356] "Generic (PLEG): container finished" podID="67034717-310b-4454-b9a4-c39d7c22ed22" containerID="67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade" exitCode=0
Sep 08 13:21:26.085399 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:26.085347    2578 generic.go:356] "Generic (PLEG): container finished" podID="67034717-310b-4454-b9a4-c39d7c22ed22" containerID="b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2" exitCode=0
Sep 08 13:21:26.085399 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:26.085355    2578 generic.go:356] "Generic (PLEG): container finished" podID="67034717-310b-4454-b9a4-c39d7c22ed22" containerID="9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c" exitCode=0
Sep 08 13:21:26.085399 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:26.085386    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"67034717-310b-4454-b9a4-c39d7c22ed22","Type":"ContainerDied","Data":"236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb"}
Sep 08 13:21:26.085724 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:26.085448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"67034717-310b-4454-b9a4-c39d7c22ed22","Type":"ContainerDied","Data":"67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade"}
Sep 08 13:21:26.085724 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:26.085474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"67034717-310b-4454-b9a4-c39d7c22ed22","Type":"ContainerDied","Data":"b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2"}
Sep 08 13:21:26.085724 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:26.085500    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"67034717-310b-4454-b9a4-c39d7c22ed22","Type":"ContainerDied","Data":"9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c"}
Sep 08 13:21:27.023030 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.023005    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.080363 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.080331    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/67034717-310b-4454-b9a4-c39d7c22ed22-config-out\") pod \"67034717-310b-4454-b9a4-c39d7c22ed22\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") "
Sep 08 13:21:27.080569 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.080399    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-web-config\") pod \"67034717-310b-4454-b9a4-c39d7c22ed22\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") "
Sep 08 13:21:27.080569 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.080433    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy-web\") pod \"67034717-310b-4454-b9a4-c39d7c22ed22\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") "
Sep 08 13:21:27.080569 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.080461    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-config-volume\") pod \"67034717-310b-4454-b9a4-c39d7c22ed22\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") "
Sep 08 13:21:27.080569 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.080484    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-cluster-tls-config\") pod \"67034717-310b-4454-b9a4-c39d7c22ed22\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") "
Sep 08 13:21:27.080569 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.080511    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/67034717-310b-4454-b9a4-c39d7c22ed22-tls-assets\") pod \"67034717-310b-4454-b9a4-c39d7c22ed22\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") "
Sep 08 13:21:27.080569 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.080546    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6j7g\" (UniqueName: \"kubernetes.io/projected/67034717-310b-4454-b9a4-c39d7c22ed22-kube-api-access-r6j7g\") pod \"67034717-310b-4454-b9a4-c39d7c22ed22\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") "
Sep 08 13:21:27.080896 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.080631    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy\") pod \"67034717-310b-4454-b9a4-c39d7c22ed22\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") "
Sep 08 13:21:27.080896 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.080659    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/67034717-310b-4454-b9a4-c39d7c22ed22-alertmanager-main-db\") pod \"67034717-310b-4454-b9a4-c39d7c22ed22\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") "
Sep 08 13:21:27.080896 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.080694    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-main-tls\") pod \"67034717-310b-4454-b9a4-c39d7c22ed22\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") "
Sep 08 13:21:27.080896 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.080734    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy-metric\") pod \"67034717-310b-4454-b9a4-c39d7c22ed22\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") "
Sep 08 13:21:27.080896 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.080767    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67034717-310b-4454-b9a4-c39d7c22ed22-alertmanager-trusted-ca-bundle\") pod \"67034717-310b-4454-b9a4-c39d7c22ed22\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") "
Sep 08 13:21:27.080896 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.080797    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67034717-310b-4454-b9a4-c39d7c22ed22-metrics-client-ca\") pod \"67034717-310b-4454-b9a4-c39d7c22ed22\" (UID: \"67034717-310b-4454-b9a4-c39d7c22ed22\") "
Sep 08 13:21:27.081296 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.081263    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67034717-310b-4454-b9a4-c39d7c22ed22-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "67034717-310b-4454-b9a4-c39d7c22ed22" (UID: "67034717-310b-4454-b9a4-c39d7c22ed22"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:21:27.082789 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.082147    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67034717-310b-4454-b9a4-c39d7c22ed22-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "67034717-310b-4454-b9a4-c39d7c22ed22" (UID: "67034717-310b-4454-b9a4-c39d7c22ed22"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:27.084794 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.084763    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67034717-310b-4454-b9a4-c39d7c22ed22-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "67034717-310b-4454-b9a4-c39d7c22ed22" (UID: "67034717-310b-4454-b9a4-c39d7c22ed22"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:27.084963 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.084834    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "67034717-310b-4454-b9a4-c39d7c22ed22" (UID: "67034717-310b-4454-b9a4-c39d7c22ed22"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:27.086897 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.086823    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67034717-310b-4454-b9a4-c39d7c22ed22-config-out" (OuterVolumeSpecName: "config-out") pod "67034717-310b-4454-b9a4-c39d7c22ed22" (UID: "67034717-310b-4454-b9a4-c39d7c22ed22"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:21:27.087007 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.086948    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67034717-310b-4454-b9a4-c39d7c22ed22-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "67034717-310b-4454-b9a4-c39d7c22ed22" (UID: "67034717-310b-4454-b9a4-c39d7c22ed22"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:21:27.087835 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.087807    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-config-volume" (OuterVolumeSpecName: "config-volume") pod "67034717-310b-4454-b9a4-c39d7c22ed22" (UID: "67034717-310b-4454-b9a4-c39d7c22ed22"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:27.088002 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.087981    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67034717-310b-4454-b9a4-c39d7c22ed22-kube-api-access-r6j7g" (OuterVolumeSpecName: "kube-api-access-r6j7g") pod "67034717-310b-4454-b9a4-c39d7c22ed22" (UID: "67034717-310b-4454-b9a4-c39d7c22ed22"). InnerVolumeSpecName "kube-api-access-r6j7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:21:27.088377 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.088347    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "67034717-310b-4454-b9a4-c39d7c22ed22" (UID: "67034717-310b-4454-b9a4-c39d7c22ed22"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:27.088745 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.088707    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "67034717-310b-4454-b9a4-c39d7c22ed22" (UID: "67034717-310b-4454-b9a4-c39d7c22ed22"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:27.089570 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.089535    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "67034717-310b-4454-b9a4-c39d7c22ed22" (UID: "67034717-310b-4454-b9a4-c39d7c22ed22"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:27.094100 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.094070    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "67034717-310b-4454-b9a4-c39d7c22ed22" (UID: "67034717-310b-4454-b9a4-c39d7c22ed22"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:27.096165 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.096137    2578 generic.go:356] "Generic (PLEG): container finished" podID="67034717-310b-4454-b9a4-c39d7c22ed22" containerID="e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d" exitCode=0
Sep 08 13:21:27.096165 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.096165    2578 generic.go:356] "Generic (PLEG): container finished" podID="67034717-310b-4454-b9a4-c39d7c22ed22" containerID="be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1" exitCode=0
Sep 08 13:21:27.096308 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.096196    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"67034717-310b-4454-b9a4-c39d7c22ed22","Type":"ContainerDied","Data":"e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d"}
Sep 08 13:21:27.096308 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.096239    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.096308 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.096275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"67034717-310b-4454-b9a4-c39d7c22ed22","Type":"ContainerDied","Data":"be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1"}
Sep 08 13:21:27.096438 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.096321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"67034717-310b-4454-b9a4-c39d7c22ed22","Type":"ContainerDied","Data":"c6ae6283e3e2e0fe18474b0b4c952020d508124357dfab41af46f4cb09551873"}
Sep 08 13:21:27.096438 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.096343    2578 scope.go:117] "RemoveContainer" containerID="236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb"
Sep 08 13:21:27.101126 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.101095    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-web-config" (OuterVolumeSpecName: "web-config") pod "67034717-310b-4454-b9a4-c39d7c22ed22" (UID: "67034717-310b-4454-b9a4-c39d7c22ed22"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:27.104867 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.104846    2578 scope.go:117] "RemoveContainer" containerID="e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d"
Sep 08 13:21:27.111668 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.111604    2578 scope.go:117] "RemoveContainer" containerID="67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade"
Sep 08 13:21:27.118562 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.118529    2578 scope.go:117] "RemoveContainer" containerID="be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1"
Sep 08 13:21:27.125768 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.125750    2578 scope.go:117] "RemoveContainer" containerID="b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2"
Sep 08 13:21:27.132208 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.132186    2578 scope.go:117] "RemoveContainer" containerID="9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c"
Sep 08 13:21:27.138809 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.138793    2578 scope.go:117] "RemoveContainer" containerID="ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420"
Sep 08 13:21:27.145268 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.145252    2578 scope.go:117] "RemoveContainer" containerID="236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb"
Sep 08 13:21:27.145545 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:21:27.145525    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb\": container with ID starting with 236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb not found: ID does not exist" containerID="236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb"
Sep 08 13:21:27.145596 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.145553    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb"} err="failed to get container status \"236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb\": rpc error: code = NotFound desc = could not find container \"236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb\": container with ID starting with 236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb not found: ID does not exist"
Sep 08 13:21:27.145596 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.145574    2578 scope.go:117] "RemoveContainer" containerID="e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d"
Sep 08 13:21:27.145863 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:21:27.145846    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d\": container with ID starting with e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d not found: ID does not exist" containerID="e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d"
Sep 08 13:21:27.145900 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.145870    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d"} err="failed to get container status \"e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d\": rpc error: code = NotFound desc = could not find container \"e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d\": container with ID starting with e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d not found: ID does not exist"
Sep 08 13:21:27.145900 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.145886    2578 scope.go:117] "RemoveContainer" containerID="67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade"
Sep 08 13:21:27.146096 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:21:27.146078    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade\": container with ID starting with 67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade not found: ID does not exist" containerID="67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade"
Sep 08 13:21:27.146136 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.146101    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade"} err="failed to get container status \"67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade\": rpc error: code = NotFound desc = could not find container \"67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade\": container with ID starting with 67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade not found: ID does not exist"
Sep 08 13:21:27.146136 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.146118    2578 scope.go:117] "RemoveContainer" containerID="be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1"
Sep 08 13:21:27.146319 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:21:27.146304    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1\": container with ID starting with be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1 not found: ID does not exist" containerID="be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1"
Sep 08 13:21:27.146357 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.146322    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1"} err="failed to get container status \"be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1\": rpc error: code = NotFound desc = could not find container \"be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1\": container with ID starting with be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1 not found: ID does not exist"
Sep 08 13:21:27.146357 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.146334    2578 scope.go:117] "RemoveContainer" containerID="b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2"
Sep 08 13:21:27.146531 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:21:27.146512    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2\": container with ID starting with b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2 not found: ID does not exist" containerID="b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2"
Sep 08 13:21:27.146597 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.146539    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2"} err="failed to get container status \"b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2\": rpc error: code = NotFound desc = could not find container \"b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2\": container with ID starting with b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2 not found: ID does not exist"
Sep 08 13:21:27.146597 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.146561    2578 scope.go:117] "RemoveContainer" containerID="9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c"
Sep 08 13:21:27.146821 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:21:27.146803    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c\": container with ID starting with 9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c not found: ID does not exist" containerID="9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c"
Sep 08 13:21:27.146862 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.146825    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c"} err="failed to get container status \"9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c\": rpc error: code = NotFound desc = could not find container \"9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c\": container with ID starting with 9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c not found: ID does not exist"
Sep 08 13:21:27.146862 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.146837    2578 scope.go:117] "RemoveContainer" containerID="ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420"
Sep 08 13:21:27.147053 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:21:27.147038    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420\": container with ID starting with ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420 not found: ID does not exist" containerID="ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420"
Sep 08 13:21:27.147090 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.147058    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420"} err="failed to get container status \"ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420\": rpc error: code = NotFound desc = could not find container \"ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420\": container with ID starting with ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420 not found: ID does not exist"
Sep 08 13:21:27.147090 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.147072    2578 scope.go:117] "RemoveContainer" containerID="236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb"
Sep 08 13:21:27.147308 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.147294    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb"} err="failed to get container status \"236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb\": rpc error: code = NotFound desc = could not find container \"236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb\": container with ID starting with 236b931eae484e05f07b2dc66fa09a64c4a25613ad69695633d168b324ad23bb not found: ID does not exist"
Sep 08 13:21:27.147345 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.147309    2578 scope.go:117] "RemoveContainer" containerID="e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d"
Sep 08 13:21:27.147509 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.147492    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d"} err="failed to get container status \"e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d\": rpc error: code = NotFound desc = could not find container \"e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d\": container with ID starting with e8b8f0f3e1d6b0d249ac945e059f13014cd112e6e07508146fd4d434aa341a8d not found: ID does not exist"
Sep 08 13:21:27.147554 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.147511    2578 scope.go:117] "RemoveContainer" containerID="67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade"
Sep 08 13:21:27.147773 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.147755    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade"} err="failed to get container status \"67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade\": rpc error: code = NotFound desc = could not find container \"67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade\": container with ID starting with 67b2cb071a1247fca3ebd1b5a9aa388c44fa986b4911cd718bd8f59cb52f2ade not found: ID does not exist"
Sep 08 13:21:27.147817 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.147777    2578 scope.go:117] "RemoveContainer" containerID="be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1"
Sep 08 13:21:27.147991 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.147974    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1"} err="failed to get container status \"be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1\": rpc error: code = NotFound desc = could not find container \"be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1\": container with ID starting with be0279a334f1ed7faed752da0ca93287c88c3dd96fbe95842668fa51bf98ede1 not found: ID does not exist"
Sep 08 13:21:27.148034 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.147991    2578 scope.go:117] "RemoveContainer" containerID="b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2"
Sep 08 13:21:27.148193 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.148178    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2"} err="failed to get container status \"b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2\": rpc error: code = NotFound desc = could not find container \"b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2\": container with ID starting with b7f704f3974c87db3935dd0aba1536707b55d4fab95cad3bef3ccad23204d8b2 not found: ID does not exist"
Sep 08 13:21:27.148234 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.148195    2578 scope.go:117] "RemoveContainer" containerID="9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c"
Sep 08 13:21:27.148395 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.148380    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c"} err="failed to get container status \"9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c\": rpc error: code = NotFound desc = could not find container \"9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c\": container with ID starting with 9dce8f91098f6577c901656b3a4b4318be193001b7b9d268bcb31a1455d4720c not found: ID does not exist"
Sep 08 13:21:27.148440 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.148395    2578 scope.go:117] "RemoveContainer" containerID="ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420"
Sep 08 13:21:27.148688 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.148607    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420"} err="failed to get container status \"ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420\": rpc error: code = NotFound desc = could not find container \"ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420\": container with ID starting with ac8c85b41290d08bbb05038d6d15c419995144174160a3bb1f47123338184420 not found: ID does not exist"
Sep 08 13:21:27.182052 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.182030    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67034717-310b-4454-b9a4-c39d7c22ed22-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:27.182117 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.182052    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67034717-310b-4454-b9a4-c39d7c22ed22-metrics-client-ca\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:27.182117 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.182063    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/67034717-310b-4454-b9a4-c39d7c22ed22-config-out\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:27.182117 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.182072    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-web-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:27.182117 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.182080    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:27.182117 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.182088    2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-config-volume\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:27.182117 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.182098    2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-cluster-tls-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:27.182117 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.182111    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/67034717-310b-4454-b9a4-c39d7c22ed22-tls-assets\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:27.182117 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.182120    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6j7g\" (UniqueName: \"kubernetes.io/projected/67034717-310b-4454-b9a4-c39d7c22ed22-kube-api-access-r6j7g\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:27.182353 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.182128    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:27.182353 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.182137    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/67034717-310b-4454-b9a4-c39d7c22ed22-alertmanager-main-db\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:27.182353 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.182147    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-main-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:27.182353 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.182155    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/67034717-310b-4454-b9a4-c39d7c22ed22-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:27.421515 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.421483    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:21:27.427202 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.427178    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:21:27.463998 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.463972    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:21:27.464260 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464247    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="prom-label-proxy"
Sep 08 13:21:27.464299 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464262    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="prom-label-proxy"
Sep 08 13:21:27.464299 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464271    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="kube-rbac-proxy-web"
Sep 08 13:21:27.464299 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464276    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="kube-rbac-proxy-web"
Sep 08 13:21:27.464299 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464284    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="config-reloader"
Sep 08 13:21:27.464299 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464290    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="config-reloader"
Sep 08 13:21:27.464299 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464298    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="kube-rbac-proxy"
Sep 08 13:21:27.464465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464303    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="kube-rbac-proxy"
Sep 08 13:21:27.464465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464310    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="init-config-reloader"
Sep 08 13:21:27.464465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464315    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="init-config-reloader"
Sep 08 13:21:27.464465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464322    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="kube-rbac-proxy-metric"
Sep 08 13:21:27.464465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464327    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="kube-rbac-proxy-metric"
Sep 08 13:21:27.464465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464337    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="alertmanager"
Sep 08 13:21:27.464465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464342    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="alertmanager"
Sep 08 13:21:27.464465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464387    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="kube-rbac-proxy"
Sep 08 13:21:27.464465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464394    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="kube-rbac-proxy-web"
Sep 08 13:21:27.464465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464401    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="config-reloader"
Sep 08 13:21:27.464465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464406    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="alertmanager"
Sep 08 13:21:27.464465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464412    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="kube-rbac-proxy-metric"
Sep 08 13:21:27.464465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.464417    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" containerName="prom-label-proxy"
Sep 08 13:21:27.469365 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.469341    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.473425 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.473405    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 08 13:21:27.473751 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.473727    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 08 13:21:27.473751 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.473744    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 08 13:21:27.473973 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.473957    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 08 13:21:27.474047 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.473975    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 08 13:21:27.474516 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.474499    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 08 13:21:27.474604 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.474582    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 08 13:21:27.474769 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.474747    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8hstd\""
Sep 08 13:21:27.475240 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.475214    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 08 13:21:27.480877 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.480859    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 08 13:21:27.486848 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.486822    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:21:27.585153 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.585114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.585153 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.585153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-config-volume\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.585348 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.585181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.585348 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.585207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.585348 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.585229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.585348 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.585256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-web-config\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.585472 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.585381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.585472 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.585409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.585472 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.585428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-config-out\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.585472 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.585445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.585862 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.585511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms62l\" (UniqueName: \"kubernetes.io/projected/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-kube-api-access-ms62l\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.585862 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.585529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.585862 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.585560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.686909 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.686827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-web-config\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.686909 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.686897    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.687089 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.686924    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.687089 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.686946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-config-out\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.687089 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.686970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.687089 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.687007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms62l\" (UniqueName: \"kubernetes.io/projected/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-kube-api-access-ms62l\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.687089 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.687031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.687089 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.687072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.687390 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.687109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.687390 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.687133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-config-volume\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.687390 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.687160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.687390 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.687183    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.687390 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.687212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.687390 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.687384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.688362 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.688033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.690105 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.690071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.690212 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.690110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.690212 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.690130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.690212 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.690142    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.690212 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.690148    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.690419 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.690218    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-web-config\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.690740 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.690722    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-config-volume\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.690789 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.690735    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-config-out\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.690988 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.690971    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.691923 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.691908    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.695534 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.695518    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms62l\" (UniqueName: \"kubernetes.io/projected/3376cf7e-4975-4688-a4ed-b5ac4acdf9f8-kube-api-access-ms62l\") pod \"alertmanager-main-0\" (UID: \"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:27.779842 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:27.779815    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:28.062176 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:28.062074    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:21:28.066138 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:21:28.066112    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3376cf7e_4975_4688_a4ed_b5ac4acdf9f8.slice/crio-9ffe5b70d7e7d13c82faee0e32b4aba19bc44cd03ad366bc2ccd6eed2493f909 WatchSource:0}: Error finding container 9ffe5b70d7e7d13c82faee0e32b4aba19bc44cd03ad366bc2ccd6eed2493f909: Status 404 returned error can't find the container with id 9ffe5b70d7e7d13c82faee0e32b4aba19bc44cd03ad366bc2ccd6eed2493f909
Sep 08 13:21:28.102809 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:28.102779    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8","Type":"ContainerStarted","Data":"9ffe5b70d7e7d13c82faee0e32b4aba19bc44cd03ad366bc2ccd6eed2493f909"}
Sep 08 13:21:29.012722 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:29.012686    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67034717-310b-4454-b9a4-c39d7c22ed22" path="/var/lib/kubelet/pods/67034717-310b-4454-b9a4-c39d7c22ed22/volumes"
Sep 08 13:21:29.108112 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:29.108023    2578 generic.go:356] "Generic (PLEG): container finished" podID="3376cf7e-4975-4688-a4ed-b5ac4acdf9f8" containerID="f199fb84a401da6507e0e44e54fd984ea9784212e39a3110ce523f1a968bafe1" exitCode=0
Sep 08 13:21:29.108112 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:29.108080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8","Type":"ContainerDied","Data":"f199fb84a401da6507e0e44e54fd984ea9784212e39a3110ce523f1a968bafe1"}
Sep 08 13:21:30.046614 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.046583    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:21:30.046905 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.046868    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="prometheus" containerID="cri-o://c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28" gracePeriod=600
Sep 08 13:21:30.046994 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.046906    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="thanos-sidecar" containerID="cri-o://fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325" gracePeriod=600
Sep 08 13:21:30.046994 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.046921    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="kube-rbac-proxy-web" containerID="cri-o://c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e" gracePeriod=600
Sep 08 13:21:30.046994 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.046939    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="config-reloader" containerID="cri-o://c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175" gracePeriod=600
Sep 08 13:21:30.046994 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.046881    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="kube-rbac-proxy" containerID="cri-o://6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be" gracePeriod=600
Sep 08 13:21:30.047173 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.046909    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="kube-rbac-proxy-thanos" containerID="cri-o://da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420" gracePeriod=600
Sep 08 13:21:30.117929 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.117895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8","Type":"ContainerStarted","Data":"7a6258b5ec0219929d63526c03140a09637b0a5bee19f31f1020e93a25643d2d"}
Sep 08 13:21:30.118322 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.117986    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8","Type":"ContainerStarted","Data":"24ce4d8dfb0b98bf2e4f0fd3647d2ebc4a3a69c7b293e160222827710479c433"}
Sep 08 13:21:30.118322 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.118051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8","Type":"ContainerStarted","Data":"437816b1810d53e2dd463441deb599d074711909bc9d4649ed295f139353b614"}
Sep 08 13:21:30.118322 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.118093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8","Type":"ContainerStarted","Data":"e67df2d9a88929f663892a961fe7e58e9a20bfdd3e213d7bc2c759cd5937f17f"}
Sep 08 13:21:30.118322 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.118128    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8","Type":"ContainerStarted","Data":"749b3ed7a34bdd311bd4bb55bc593028eeb4fd6c797892c41117b1fb20dc4927"}
Sep 08 13:21:30.118322 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.118164    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3376cf7e-4975-4688-a4ed-b5ac4acdf9f8","Type":"ContainerStarted","Data":"4f9ccda59fa816a7404a46da692cf369d4651dc5d35c14492379a5c3cd1f8418"}
Sep 08 13:21:30.147742 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.147685    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.13076397 podStartE2EDuration="3.147667243s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.903273ms" isStatefulPod=false podCreationTimestamp="2026-09-08 13:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:21:30.146050731 +0000 UTC m=+307.746148362" watchObservedRunningTime="2026-09-08 13:21:30.147667243 +0000 UTC m=+307.747764860"
Sep 08 13:21:30.303016 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.302947    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:30.411724 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.411690    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-serving-certs-ca-bundle\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.411910 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.411730    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-tls\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.411910 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.411769    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-0\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.411910 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.411862    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5171ade-f444-4e08-8975-2632ddf7cb72-tls-assets\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.411910 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.411893    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5171ade-f444-4e08-8975-2632ddf7cb72-config-out\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412147 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.411915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-db\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412147 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.411944    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412147 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.411980    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-metrics-client-ca\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412147 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412034    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-grpc-tls\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412147 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412059    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:30.412147 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412086    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412147 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412119    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-kubelet-serving-ca-bundle\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412147 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412143    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-trusted-ca-bundle\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412592 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412172    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-1\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412592 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412195    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-kube-rbac-proxy\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412592 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412265    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-2\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412592 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412290    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-web-config\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412592 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412352    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwb8n\" (UniqueName: \"kubernetes.io/projected/a5171ade-f444-4e08-8975-2632ddf7cb72-kube-api-access-kwb8n\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412592 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412351    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:30.412592 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412389    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-config\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412592 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412416    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-metrics-client-certs\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412592 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412452    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-thanos-prometheus-http-client-file\") pod \"a5171ade-f444-4e08-8975-2632ddf7cb72\" (UID: \"a5171ade-f444-4e08-8975-2632ddf7cb72\") "
Sep 08 13:21:30.412592 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412572    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:30.413094 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412752    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.413094 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412772    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-metrics-client-ca\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.413094 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.412788    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.413912 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.413589    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:21:30.414238 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.414213    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:30.414373 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.414324    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5171ade-f444-4e08-8975-2632ddf7cb72-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:21:30.414470 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.414389    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5171ade-f444-4e08-8975-2632ddf7cb72-config-out" (OuterVolumeSpecName: "config-out") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:21:30.417222 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.414692    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:30.417222 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.415226    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:30.417222 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.415355    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:30.417222 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.415610    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:30.417222 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.415932    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:30.417222 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.416877    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:30.417222 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.416935    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-config" (OuterVolumeSpecName: "config") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:30.417222 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.417009    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:30.417222 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.417029    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:30.417740 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.417260    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:30.418817 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.418795    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:30.419525 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.419502    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5171ade-f444-4e08-8975-2632ddf7cb72-kube-api-access-kwb8n" (OuterVolumeSpecName: "kube-api-access-kwb8n") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "kube-api-access-kwb8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:21:30.428899 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.428873    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-web-config" (OuterVolumeSpecName: "web-config") pod "a5171ade-f444-4e08-8975-2632ddf7cb72" (UID: "a5171ade-f444-4e08-8975-2632ddf7cb72"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:30.514031 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.513989    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwb8n\" (UniqueName: \"kubernetes.io/projected/a5171ade-f444-4e08-8975-2632ddf7cb72-kube-api-access-kwb8n\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514031 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514027    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514031 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514038    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-metrics-client-certs\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514048    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514058    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514067    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514075    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5171ade-f444-4e08-8975-2632ddf7cb72-tls-assets\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514083    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5171ade-f444-4e08-8975-2632ddf7cb72-config-out\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514090    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-db\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514098    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514107    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-grpc-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514116    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514124    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514133    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514143    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-secret-kube-rbac-proxy\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514152    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5171ade-f444-4e08-8975-2632ddf7cb72-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:30.514209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:30.514160    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5171ade-f444-4e08-8975-2632ddf7cb72-web-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:21:31.124511 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.124432    2578 generic.go:356] "Generic (PLEG): container finished" podID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerID="da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420" exitCode=0
Sep 08 13:21:31.124511 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.124458    2578 generic.go:356] "Generic (PLEG): container finished" podID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerID="6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be" exitCode=0
Sep 08 13:21:31.124511 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.124465    2578 generic.go:356] "Generic (PLEG): container finished" podID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerID="c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e" exitCode=0
Sep 08 13:21:31.124511 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.124470    2578 generic.go:356] "Generic (PLEG): container finished" podID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerID="fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325" exitCode=0
Sep 08 13:21:31.124511 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.124476    2578 generic.go:356] "Generic (PLEG): container finished" podID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerID="c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175" exitCode=0
Sep 08 13:21:31.124511 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.124482    2578 generic.go:356] "Generic (PLEG): container finished" podID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerID="c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28" exitCode=0
Sep 08 13:21:31.124511 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.124472    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5171ade-f444-4e08-8975-2632ddf7cb72","Type":"ContainerDied","Data":"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"}
Sep 08 13:21:31.124511 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.124511    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.125190 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.124550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5171ade-f444-4e08-8975-2632ddf7cb72","Type":"ContainerDied","Data":"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"}
Sep 08 13:21:31.125190 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.124585    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5171ade-f444-4e08-8975-2632ddf7cb72","Type":"ContainerDied","Data":"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"}
Sep 08 13:21:31.125190 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.124613    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5171ade-f444-4e08-8975-2632ddf7cb72","Type":"ContainerDied","Data":"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"}
Sep 08 13:21:31.125190 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.124632    2578 scope.go:117] "RemoveContainer" containerID="da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"
Sep 08 13:21:31.125190 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.124664    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5171ade-f444-4e08-8975-2632ddf7cb72","Type":"ContainerDied","Data":"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"}
Sep 08 13:21:31.125190 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.124692    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5171ade-f444-4e08-8975-2632ddf7cb72","Type":"ContainerDied","Data":"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"}
Sep 08 13:21:31.125190 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.124719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5171ade-f444-4e08-8975-2632ddf7cb72","Type":"ContainerDied","Data":"3cae3651026e6343bcaaf8753812cb85c89663b43c58fa1d2cc20726ac5a27f2"}
Sep 08 13:21:31.134480 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.133390    2578 scope.go:117] "RemoveContainer" containerID="6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"
Sep 08 13:21:31.140911 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.140882    2578 scope.go:117] "RemoveContainer" containerID="c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"
Sep 08 13:21:31.147577 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.147553    2578 scope.go:117] "RemoveContainer" containerID="fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"
Sep 08 13:21:31.149287 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.149269    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:21:31.154763 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.154741    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:21:31.155755 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.155740    2578 scope.go:117] "RemoveContainer" containerID="c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"
Sep 08 13:21:31.162313 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.162211    2578 scope.go:117] "RemoveContainer" containerID="c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"
Sep 08 13:21:31.170304 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.170285    2578 scope.go:117] "RemoveContainer" containerID="b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694"
Sep 08 13:21:31.177640 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.177596    2578 scope.go:117] "RemoveContainer" containerID="da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"
Sep 08 13:21:31.177924 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:21:31.177903    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420\": container with ID starting with da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420 not found: ID does not exist" containerID="da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"
Sep 08 13:21:31.178004 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.177931    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"} err="failed to get container status \"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420\": rpc error: code = NotFound desc = could not find container \"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420\": container with ID starting with da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420 not found: ID does not exist"
Sep 08 13:21:31.178004 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.177949    2578 scope.go:117] "RemoveContainer" containerID="6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"
Sep 08 13:21:31.178463 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:21:31.178420    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be\": container with ID starting with 6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be not found: ID does not exist" containerID="6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"
Sep 08 13:21:31.178463 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.178457    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"} err="failed to get container status \"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be\": rpc error: code = NotFound desc = could not find container \"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be\": container with ID starting with 6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be not found: ID does not exist"
Sep 08 13:21:31.178710 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.178485    2578 scope.go:117] "RemoveContainer" containerID="c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"
Sep 08 13:21:31.179186 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:21:31.179095    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e\": container with ID starting with c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e not found: ID does not exist" containerID="c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"
Sep 08 13:21:31.179186 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.179127    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"} err="failed to get container status \"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e\": rpc error: code = NotFound desc = could not find container \"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e\": container with ID starting with c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e not found: ID does not exist"
Sep 08 13:21:31.179186 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.179148    2578 scope.go:117] "RemoveContainer" containerID="fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"
Sep 08 13:21:31.179497 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:21:31.179476    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325\": container with ID starting with fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325 not found: ID does not exist" containerID="fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"
Sep 08 13:21:31.179543 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.179506    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"} err="failed to get container status \"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325\": rpc error: code = NotFound desc = could not find container \"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325\": container with ID starting with fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325 not found: ID does not exist"
Sep 08 13:21:31.179543 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.179525    2578 scope.go:117] "RemoveContainer" containerID="c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"
Sep 08 13:21:31.179809 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:21:31.179791    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175\": container with ID starting with c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175 not found: ID does not exist" containerID="c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"
Sep 08 13:21:31.179875 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.179815    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"} err="failed to get container status \"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175\": rpc error: code = NotFound desc = could not find container \"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175\": container with ID starting with c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175 not found: ID does not exist"
Sep 08 13:21:31.179875 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.179832    2578 scope.go:117] "RemoveContainer" containerID="c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"
Sep 08 13:21:31.180143 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:21:31.180127    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28\": container with ID starting with c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28 not found: ID does not exist" containerID="c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"
Sep 08 13:21:31.180187 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180147    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"} err="failed to get container status \"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28\": rpc error: code = NotFound desc = could not find container \"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28\": container with ID starting with c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28 not found: ID does not exist"
Sep 08 13:21:31.180187 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180163    2578 scope.go:117] "RemoveContainer" containerID="b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694"
Sep 08 13:21:31.180280 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180261    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:21:31.180426 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:21:31.180409    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694\": container with ID starting with b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694 not found: ID does not exist" containerID="b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694"
Sep 08 13:21:31.180465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180429    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694"} err="failed to get container status \"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694\": rpc error: code = NotFound desc = could not find container \"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694\": container with ID starting with b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694 not found: ID does not exist"
Sep 08 13:21:31.180465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180442    2578 scope.go:117] "RemoveContainer" containerID="da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"
Sep 08 13:21:31.180558 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180546    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="config-reloader"
Sep 08 13:21:31.180590 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180560    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="config-reloader"
Sep 08 13:21:31.180590 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180569    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="thanos-sidecar"
Sep 08 13:21:31.180590 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180576    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="thanos-sidecar"
Sep 08 13:21:31.180590 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180587    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="kube-rbac-proxy-thanos"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180594    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="kube-rbac-proxy-thanos"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180608    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="prometheus"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180634    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="prometheus"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180646    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="init-config-reloader"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180652    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="init-config-reloader"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180657    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="kube-rbac-proxy-web"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180662    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="kube-rbac-proxy-web"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180671    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="kube-rbac-proxy"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180676    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="kube-rbac-proxy"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180703    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"} err="failed to get container status \"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420\": rpc error: code = NotFound desc = could not find container \"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420\": container with ID starting with da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420 not found: ID does not exist"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180725    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="config-reloader"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180729    2578 scope.go:117] "RemoveContainer" containerID="6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180737    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="prometheus"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180745    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="kube-rbac-proxy-web"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180754    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="kube-rbac-proxy-thanos"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180761    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="thanos-sidecar"
Sep 08 13:21:31.180777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180768    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" containerName="kube-rbac-proxy"
Sep 08 13:21:31.181407 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.180985    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"} err="failed to get container status \"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be\": rpc error: code = NotFound desc = could not find container \"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be\": container with ID starting with 6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be not found: ID does not exist"
Sep 08 13:21:31.181407 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.181003    2578 scope.go:117] "RemoveContainer" containerID="c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"
Sep 08 13:21:31.181407 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.181279    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"} err="failed to get container status \"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e\": rpc error: code = NotFound desc = could not find container \"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e\": container with ID starting with c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e not found: ID does not exist"
Sep 08 13:21:31.181407 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.181298    2578 scope.go:117] "RemoveContainer" containerID="fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"
Sep 08 13:21:31.181866 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.181830    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"} err="failed to get container status \"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325\": rpc error: code = NotFound desc = could not find container \"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325\": container with ID starting with fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325 not found: ID does not exist"
Sep 08 13:21:31.181972 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.181864    2578 scope.go:117] "RemoveContainer" containerID="c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"
Sep 08 13:21:31.182192 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.182171    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"} err="failed to get container status \"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175\": rpc error: code = NotFound desc = could not find container \"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175\": container with ID starting with c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175 not found: ID does not exist"
Sep 08 13:21:31.182192 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.182192    2578 scope.go:117] "RemoveContainer" containerID="c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"
Sep 08 13:21:31.182468 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.182448    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"} err="failed to get container status \"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28\": rpc error: code = NotFound desc = could not find container \"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28\": container with ID starting with c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28 not found: ID does not exist"
Sep 08 13:21:31.182540 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.182471    2578 scope.go:117] "RemoveContainer" containerID="b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694"
Sep 08 13:21:31.182818 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.182768    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694"} err="failed to get container status \"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694\": rpc error: code = NotFound desc = could not find container \"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694\": container with ID starting with b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694 not found: ID does not exist"
Sep 08 13:21:31.182887 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.182817    2578 scope.go:117] "RemoveContainer" containerID="da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"
Sep 08 13:21:31.183226 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.183201    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"} err="failed to get container status \"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420\": rpc error: code = NotFound desc = could not find container \"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420\": container with ID starting with da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420 not found: ID does not exist"
Sep 08 13:21:31.183341 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.183235    2578 scope.go:117] "RemoveContainer" containerID="6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"
Sep 08 13:21:31.183527 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.183503    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"} err="failed to get container status \"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be\": rpc error: code = NotFound desc = could not find container \"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be\": container with ID starting with 6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be not found: ID does not exist"
Sep 08 13:21:31.183527 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.183529    2578 scope.go:117] "RemoveContainer" containerID="c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.183843    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"} err="failed to get container status \"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e\": rpc error: code = NotFound desc = could not find container \"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e\": container with ID starting with c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e not found: ID does not exist"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.183865    2578 scope.go:117] "RemoveContainer" containerID="fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.184091    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"} err="failed to get container status \"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325\": rpc error: code = NotFound desc = could not find container \"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325\": container with ID starting with fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325 not found: ID does not exist"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.184114    2578 scope.go:117] "RemoveContainer" containerID="c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.184335    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"} err="failed to get container status \"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175\": rpc error: code = NotFound desc = could not find container \"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175\": container with ID starting with c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175 not found: ID does not exist"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.184355    2578 scope.go:117] "RemoveContainer" containerID="c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.184561    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"} err="failed to get container status \"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28\": rpc error: code = NotFound desc = could not find container \"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28\": container with ID starting with c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28 not found: ID does not exist"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.184580    2578 scope.go:117] "RemoveContainer" containerID="b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.184932    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694"} err="failed to get container status \"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694\": rpc error: code = NotFound desc = could not find container \"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694\": container with ID starting with b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694 not found: ID does not exist"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.184953    2578 scope.go:117] "RemoveContainer" containerID="da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.185173    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"} err="failed to get container status \"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420\": rpc error: code = NotFound desc = could not find container \"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420\": container with ID starting with da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420 not found: ID does not exist"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.185192    2578 scope.go:117] "RemoveContainer" containerID="6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.185393    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"} err="failed to get container status \"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be\": rpc error: code = NotFound desc = could not find container \"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be\": container with ID starting with 6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be not found: ID does not exist"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.185411    2578 scope.go:117] "RemoveContainer" containerID="c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.185639    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"} err="failed to get container status \"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e\": rpc error: code = NotFound desc = could not find container \"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e\": container with ID starting with c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e not found: ID does not exist"
Sep 08 13:21:31.185739 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.185660    2578 scope.go:117] "RemoveContainer" containerID="fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"
Sep 08 13:21:31.186347 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.185933    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"} err="failed to get container status \"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325\": rpc error: code = NotFound desc = could not find container \"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325\": container with ID starting with fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325 not found: ID does not exist"
Sep 08 13:21:31.186347 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.185954    2578 scope.go:117] "RemoveContainer" containerID="c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"
Sep 08 13:21:31.186347 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.186194    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"} err="failed to get container status \"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175\": rpc error: code = NotFound desc = could not find container \"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175\": container with ID starting with c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175 not found: ID does not exist"
Sep 08 13:21:31.186347 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.186208    2578 scope.go:117] "RemoveContainer" containerID="c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"
Sep 08 13:21:31.186477 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.186440    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"} err="failed to get container status \"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28\": rpc error: code = NotFound desc = could not find container \"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28\": container with ID starting with c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28 not found: ID does not exist"
Sep 08 13:21:31.186477 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.186455    2578 scope.go:117] "RemoveContainer" containerID="b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694"
Sep 08 13:21:31.186734 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.186707    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694"} err="failed to get container status \"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694\": rpc error: code = NotFound desc = could not find container \"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694\": container with ID starting with b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694 not found: ID does not exist"
Sep 08 13:21:31.186734 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.186734    2578 scope.go:117] "RemoveContainer" containerID="da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"
Sep 08 13:21:31.186960 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.186943    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"} err="failed to get container status \"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420\": rpc error: code = NotFound desc = could not find container \"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420\": container with ID starting with da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420 not found: ID does not exist"
Sep 08 13:21:31.187009 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.186960    2578 scope.go:117] "RemoveContainer" containerID="6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"
Sep 08 13:21:31.187209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.187186    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"} err="failed to get container status \"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be\": rpc error: code = NotFound desc = could not find container \"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be\": container with ID starting with 6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be not found: ID does not exist"
Sep 08 13:21:31.187279 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.187211    2578 scope.go:117] "RemoveContainer" containerID="c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"
Sep 08 13:21:31.187279 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.187247    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.187499 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.187480    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"} err="failed to get container status \"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e\": rpc error: code = NotFound desc = could not find container \"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e\": container with ID starting with c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e not found: ID does not exist"
Sep 08 13:21:31.187538 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.187501    2578 scope.go:117] "RemoveContainer" containerID="fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"
Sep 08 13:21:31.187802 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.187772    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"} err="failed to get container status \"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325\": rpc error: code = NotFound desc = could not find container \"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325\": container with ID starting with fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325 not found: ID does not exist"
Sep 08 13:21:31.187884 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.187803    2578 scope.go:117] "RemoveContainer" containerID="c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"
Sep 08 13:21:31.188109 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.188087    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"} err="failed to get container status \"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175\": rpc error: code = NotFound desc = could not find container \"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175\": container with ID starting with c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175 not found: ID does not exist"
Sep 08 13:21:31.188175 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.188111    2578 scope.go:117] "RemoveContainer" containerID="c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"
Sep 08 13:21:31.188392 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.188375    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"} err="failed to get container status \"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28\": rpc error: code = NotFound desc = could not find container \"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28\": container with ID starting with c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28 not found: ID does not exist"
Sep 08 13:21:31.188439 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.188393    2578 scope.go:117] "RemoveContainer" containerID="b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694"
Sep 08 13:21:31.188778 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.188751    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694"} err="failed to get container status \"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694\": rpc error: code = NotFound desc = could not find container \"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694\": container with ID starting with b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694 not found: ID does not exist"
Sep 08 13:21:31.188778 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.188780    2578 scope.go:117] "RemoveContainer" containerID="da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"
Sep 08 13:21:31.189091 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.189068    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420"} err="failed to get container status \"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420\": rpc error: code = NotFound desc = could not find container \"da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420\": container with ID starting with da491b6baec0fe05d3368f9793dcf4618f227dfbe022e1e32ddf404269d3c420 not found: ID does not exist"
Sep 08 13:21:31.189150 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.189094    2578 scope.go:117] "RemoveContainer" containerID="6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"
Sep 08 13:21:31.189449 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.189423    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be"} err="failed to get container status \"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be\": rpc error: code = NotFound desc = could not find container \"6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be\": container with ID starting with 6ddcbae56027a28ec5b44e5655651d8b9855238f76ff7f14c79ad44ff02f39be not found: ID does not exist"
Sep 08 13:21:31.189449 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.189445    2578 scope.go:117] "RemoveContainer" containerID="c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"
Sep 08 13:21:31.189740 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.189719    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e"} err="failed to get container status \"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e\": rpc error: code = NotFound desc = could not find container \"c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e\": container with ID starting with c53db731866e006db2832d3b6683ee834c99961ab20a5a6ca515a7317c35eb0e not found: ID does not exist"
Sep 08 13:21:31.189740 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.189739    2578 scope.go:117] "RemoveContainer" containerID="fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"
Sep 08 13:21:31.190003 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.189981    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325"} err="failed to get container status \"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325\": rpc error: code = NotFound desc = could not find container \"fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325\": container with ID starting with fb036a790ec20585c707acff80960b9bb7ec035a42a13881b0c42d7e6f48c325 not found: ID does not exist"
Sep 08 13:21:31.190077 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.190004    2578 scope.go:117] "RemoveContainer" containerID="c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"
Sep 08 13:21:31.190256 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.190233    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175"} err="failed to get container status \"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175\": rpc error: code = NotFound desc = could not find container \"c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175\": container with ID starting with c573df12cc09789b1dd7b7d53502d2a7b3b1c3626f5369e2c83f51af5cca7175 not found: ID does not exist"
Sep 08 13:21:31.190322 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.190258    2578 scope.go:117] "RemoveContainer" containerID="c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"
Sep 08 13:21:31.190543 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.190520    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28"} err="failed to get container status \"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28\": rpc error: code = NotFound desc = could not find container \"c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28\": container with ID starting with c45015d6011a046ceef703c43b1b5bee8dc134461bd39b34fe4900fe4863bd28 not found: ID does not exist"
Sep 08 13:21:31.190640 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.190545    2578 scope.go:117] "RemoveContainer" containerID="b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694"
Sep 08 13:21:31.190740 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.190718    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 08 13:21:31.190892 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.190816    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694"} err="failed to get container status \"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694\": rpc error: code = NotFound desc = could not find container \"b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694\": container with ID starting with b156aba5c39dbb6563e1131be7f68d27fe939a44ae3313f1021e06f9d8eb4694 not found: ID does not exist"
Sep 08 13:21:31.190954 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.190933    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 08 13:21:31.191171 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.191157    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 08 13:21:31.191310 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.191288    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-88dsx\""
Sep 08 13:21:31.191310 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.191307    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 08 13:21:31.191462 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.191308    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 08 13:21:31.191462 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.191358    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 08 13:21:31.191462 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.191377    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 08 13:21:31.191650 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.191596    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 08 13:21:31.191650 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.191611    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-82ngavpe2jg48\""
Sep 08 13:21:31.191650 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.191630    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 08 13:21:31.191869 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.191851    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 08 13:21:31.191933 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.191854    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 08 13:21:31.192230 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.192215    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 13:21:31.192808 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.192791    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 08 13:21:31.195857 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.195839    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 08 13:21:31.198016 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.198000    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 08 13:21:31.203121 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.203101    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:21:31.320829 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.320795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.320829 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.320828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/008bac7a-8bcb-4a02-ba76-f85249da20be-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321042 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.320861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321042 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.320925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-web-config\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321042 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.320948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321042 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.320971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321042 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.320989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321042 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.321013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321289 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.321063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/008bac7a-8bcb-4a02-ba76-f85249da20be-config-out\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321289 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.321087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-config\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321289 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.321100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321289 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.321141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321289 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.321162    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321289 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.321208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321289 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.321255    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/008bac7a-8bcb-4a02-ba76-f85249da20be-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321524 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.321291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321524 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.321312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321524 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.321337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmjn5\" (UniqueName: \"kubernetes.io/projected/008bac7a-8bcb-4a02-ba76-f85249da20be-kube-api-access-fmjn5\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321524 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.321371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.321524 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.321391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.422642 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.422583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.422814 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.422651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/008bac7a-8bcb-4a02-ba76-f85249da20be-config-out\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.422814 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.422683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-config\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.422814 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.422708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.422814 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.422755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.422814 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.422784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.423061 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.422811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.423061 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.422842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/008bac7a-8bcb-4a02-ba76-f85249da20be-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.423061 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.422873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.423061 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.422899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.423061 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.422931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmjn5\" (UniqueName: \"kubernetes.io/projected/008bac7a-8bcb-4a02-ba76-f85249da20be-kube-api-access-fmjn5\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.423061 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.422965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.423061 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.423004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.423061 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.423036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.423446 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.423065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/008bac7a-8bcb-4a02-ba76-f85249da20be-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.423446 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.423106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.423446 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.423146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-web-config\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.423446 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.423175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.423446 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.423205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.423446 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.423235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.423661 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.423595    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.424194 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.423867    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/008bac7a-8bcb-4a02-ba76-f85249da20be-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.424637 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.424590    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.426716 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.426318    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.426716 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.426640    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.426873 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.426771    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.426873 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.426836    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/008bac7a-8bcb-4a02-ba76-f85249da20be-config-out\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.427164 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.426991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.427282 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.427241    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-config\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.427571 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.427544    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.427787 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.427761    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.427846 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.427821    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.427896 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.427873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.428312 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.428283    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-web-config\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.428386 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.428362    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.428666 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.428639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.428934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.428917    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/008bac7a-8bcb-4a02-ba76-f85249da20be-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.429014 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.428982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008bac7a-8bcb-4a02-ba76-f85249da20be-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.429286 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.429268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/008bac7a-8bcb-4a02-ba76-f85249da20be-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.432309 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.432288    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmjn5\" (UniqueName: \"kubernetes.io/projected/008bac7a-8bcb-4a02-ba76-f85249da20be-kube-api-access-fmjn5\") pod \"prometheus-k8s-0\" (UID: \"008bac7a-8bcb-4a02-ba76-f85249da20be\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.497983 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.497953    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:21:31.781966 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:31.781939    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 08 13:21:31.783374 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:21:31.783348    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008bac7a_8bcb_4a02_ba76_f85249da20be.slice/crio-bea1b92c98bb7b8c08862ab438c5ac30667a6e705ca637eb3c817c1f35bd74dd WatchSource:0}: Error finding container bea1b92c98bb7b8c08862ab438c5ac30667a6e705ca637eb3c817c1f35bd74dd: Status 404 returned error can't find the container with id bea1b92c98bb7b8c08862ab438c5ac30667a6e705ca637eb3c817c1f35bd74dd
Sep 08 13:21:32.131344 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:32.131256    2578 generic.go:356] "Generic (PLEG): container finished" podID="008bac7a-8bcb-4a02-ba76-f85249da20be" containerID="00eed417f30c5e047d688ae2c414ab4cdfb8a14c78db115c7681ae6b7948b753" exitCode=0
Sep 08 13:21:32.131841 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:32.131344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"008bac7a-8bcb-4a02-ba76-f85249da20be","Type":"ContainerDied","Data":"00eed417f30c5e047d688ae2c414ab4cdfb8a14c78db115c7681ae6b7948b753"}
Sep 08 13:21:32.131919 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:32.131895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"008bac7a-8bcb-4a02-ba76-f85249da20be","Type":"ContainerStarted","Data":"bea1b92c98bb7b8c08862ab438c5ac30667a6e705ca637eb3c817c1f35bd74dd"}
Sep 08 13:21:33.013367 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:33.013333    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5171ade-f444-4e08-8975-2632ddf7cb72" path="/var/lib/kubelet/pods/a5171ade-f444-4e08-8975-2632ddf7cb72/volumes"
Sep 08 13:21:33.139144 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:33.139114    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"008bac7a-8bcb-4a02-ba76-f85249da20be","Type":"ContainerStarted","Data":"97fa193dc1ff0c66bcdd5897aef4b2573046a9fd295d1df9afc956065f203469"}
Sep 08 13:21:33.139570 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:33.139187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"008bac7a-8bcb-4a02-ba76-f85249da20be","Type":"ContainerStarted","Data":"72a6b9e2566c6309d3c7375a4538cf5d187ee3a3c26227400ea64c386ec7ef35"}
Sep 08 13:21:33.139570 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:33.139227    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"008bac7a-8bcb-4a02-ba76-f85249da20be","Type":"ContainerStarted","Data":"8e8daeb6fdbad8db88ae6bc88a6b2888b87bd32d0f408d3dc1533fe493f9a9e4"}
Sep 08 13:21:33.139570 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:33.139267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"008bac7a-8bcb-4a02-ba76-f85249da20be","Type":"ContainerStarted","Data":"d1d6b8fa2ef56021c87c377d015651ad9d6797c5ff6cb9e74b356dd164967f15"}
Sep 08 13:21:33.139570 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:33.139310    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"008bac7a-8bcb-4a02-ba76-f85249da20be","Type":"ContainerStarted","Data":"e1945c7ce0ddfd04ec4d3bc2cdc96b7c3f2b64febff7e00180ac4797db64e2ca"}
Sep 08 13:21:33.139570 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:33.139346    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"008bac7a-8bcb-4a02-ba76-f85249da20be","Type":"ContainerStarted","Data":"4133bf87168b64babb01f98174c3b98cc4daf36ef19e747d877477fbb7311af2"}
Sep 08 13:21:33.180197 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:33.180138    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.161656269 podStartE2EDuration="2.180120889s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.46462ms" isStatefulPod=false podCreationTimestamp="2026-09-08 13:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:21:33.177672702 +0000 UTC m=+310.777770348" watchObservedRunningTime="2026-09-08 13:21:33.180120889 +0000 UTC m=+310.780218514"
Sep 08 13:21:36.498446 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:21:36.498410    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:22:31.498389 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:22:31.498354    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:22:31.514404 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:22:31.514379    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:22:32.387407 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:22:32.387380    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 08 13:26:56.686100 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:56.686018    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-mwmwq"]
Sep 08 13:26:56.689178 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:56.689163    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mwmwq"
Sep 08 13:26:56.692062 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:56.692038    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 08 13:26:56.692185 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:56.692068    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 08 13:26:56.692185 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:56.692084    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 08 13:26:56.693108 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:56.693086    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 08 13:26:56.693228 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:56.693086    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-lnbgj\""
Sep 08 13:26:56.697867 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:56.697846    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-mwmwq"]
Sep 08 13:26:56.775494 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:56.775466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtjvh\" (UniqueName: \"kubernetes.io/projected/a7691dde-c1f7-4a28-9410-4b61288dc670-kube-api-access-dtjvh\") pod \"s3-init-mwmwq\" (UID: \"a7691dde-c1f7-4a28-9410-4b61288dc670\") " pod="kserve/s3-init-mwmwq"
Sep 08 13:26:56.775642 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:56.775545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7691dde-c1f7-4a28-9410-4b61288dc670-model-cache\") pod \"s3-init-mwmwq\" (UID: \"a7691dde-c1f7-4a28-9410-4b61288dc670\") " pod="kserve/s3-init-mwmwq"
Sep 08 13:26:56.876271 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:56.876241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7691dde-c1f7-4a28-9410-4b61288dc670-model-cache\") pod \"s3-init-mwmwq\" (UID: \"a7691dde-c1f7-4a28-9410-4b61288dc670\") " pod="kserve/s3-init-mwmwq"
Sep 08 13:26:56.876400 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:56.876277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtjvh\" (UniqueName: \"kubernetes.io/projected/a7691dde-c1f7-4a28-9410-4b61288dc670-kube-api-access-dtjvh\") pod \"s3-init-mwmwq\" (UID: \"a7691dde-c1f7-4a28-9410-4b61288dc670\") " pod="kserve/s3-init-mwmwq"
Sep 08 13:26:56.876592 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:56.876574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7691dde-c1f7-4a28-9410-4b61288dc670-model-cache\") pod \"s3-init-mwmwq\" (UID: \"a7691dde-c1f7-4a28-9410-4b61288dc670\") " pod="kserve/s3-init-mwmwq"
Sep 08 13:26:56.885019 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:56.885001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtjvh\" (UniqueName: \"kubernetes.io/projected/a7691dde-c1f7-4a28-9410-4b61288dc670-kube-api-access-dtjvh\") pod \"s3-init-mwmwq\" (UID: \"a7691dde-c1f7-4a28-9410-4b61288dc670\") " pod="kserve/s3-init-mwmwq"
Sep 08 13:26:57.007113 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:57.007043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mwmwq"
Sep 08 13:26:57.275474 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:57.275383    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-mwmwq"]
Sep 08 13:26:57.277814 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:26:57.277787    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7691dde_c1f7_4a28_9410_4b61288dc670.slice/crio-c2e67d93be39c82d4957e9e3866769990b8dbbc2eded584ff9c6d4b160e888eb WatchSource:0}: Error finding container c2e67d93be39c82d4957e9e3866769990b8dbbc2eded584ff9c6d4b160e888eb: Status 404 returned error can't find the container with id c2e67d93be39c82d4957e9e3866769990b8dbbc2eded584ff9c6d4b160e888eb
Sep 08 13:26:57.279841 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:57.279821    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:26:57.382515 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:26:57.382477    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mwmwq" event={"ID":"a7691dde-c1f7-4a28-9410-4b61288dc670","Type":"ContainerStarted","Data":"c2e67d93be39c82d4957e9e3866769990b8dbbc2eded584ff9c6d4b160e888eb"}
Sep 08 13:27:01.402561 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:27:01.402518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mwmwq" event={"ID":"a7691dde-c1f7-4a28-9410-4b61288dc670","Type":"ContainerStarted","Data":"802883adde65ec463b41aca96acf2d91930ac71209923558ecf09fbf647bf341"}
Sep 08 13:27:03.411958 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:27:03.411924    2578 generic.go:356] "Generic (PLEG): container finished" podID="a7691dde-c1f7-4a28-9410-4b61288dc670" containerID="802883adde65ec463b41aca96acf2d91930ac71209923558ecf09fbf647bf341" exitCode=0
Sep 08 13:27:03.412428 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:27:03.411995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mwmwq" event={"ID":"a7691dde-c1f7-4a28-9410-4b61288dc670","Type":"ContainerDied","Data":"802883adde65ec463b41aca96acf2d91930ac71209923558ecf09fbf647bf341"}
Sep 08 13:27:08.437268 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:27:08.437229    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mwmwq" event={"ID":"a7691dde-c1f7-4a28-9410-4b61288dc670","Type":"ContainerStarted","Data":"318bd2a58c6aed73cc8f476969ca48a3c4dbc07d5a72dcae3c182ffc2cebfd38"}
Sep 08 13:27:08.454550 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:27:08.454500    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-mwmwq" podStartSLOduration=2.6300343550000003 podStartE2EDuration="12.454485958s" totalImagesPullingTime="8.097879067s" totalInitContainerRuntime="1.726572536s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:26:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:27:08.453687764 +0000 UTC m=+646.053785393" watchObservedRunningTime="2026-09-08 13:27:08.454485958 +0000 UTC m=+646.054583583"
Sep 08 13:28:13.688034 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:13.687994    2578 generic.go:356] "Generic (PLEG): container finished" podID="a7691dde-c1f7-4a28-9410-4b61288dc670" containerID="318bd2a58c6aed73cc8f476969ca48a3c4dbc07d5a72dcae3c182ffc2cebfd38" exitCode=0
Sep 08 13:28:13.688427 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:13.688069    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mwmwq" event={"ID":"a7691dde-c1f7-4a28-9410-4b61288dc670","Type":"ContainerDied","Data":"318bd2a58c6aed73cc8f476969ca48a3c4dbc07d5a72dcae3c182ffc2cebfd38"}
Sep 08 13:28:14.810668 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:14.810641    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mwmwq"
Sep 08 13:28:14.932609 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:14.932575    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7691dde-c1f7-4a28-9410-4b61288dc670-model-cache\") pod \"a7691dde-c1f7-4a28-9410-4b61288dc670\" (UID: \"a7691dde-c1f7-4a28-9410-4b61288dc670\") "
Sep 08 13:28:14.932813 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:14.932682    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtjvh\" (UniqueName: \"kubernetes.io/projected/a7691dde-c1f7-4a28-9410-4b61288dc670-kube-api-access-dtjvh\") pod \"a7691dde-c1f7-4a28-9410-4b61288dc670\" (UID: \"a7691dde-c1f7-4a28-9410-4b61288dc670\") "
Sep 08 13:28:14.934849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:14.934824    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7691dde-c1f7-4a28-9410-4b61288dc670-kube-api-access-dtjvh" (OuterVolumeSpecName: "kube-api-access-dtjvh") pod "a7691dde-c1f7-4a28-9410-4b61288dc670" (UID: "a7691dde-c1f7-4a28-9410-4b61288dc670"). InnerVolumeSpecName "kube-api-access-dtjvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:28:14.954515 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:14.954461    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7691dde-c1f7-4a28-9410-4b61288dc670-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7691dde-c1f7-4a28-9410-4b61288dc670" (UID: "a7691dde-c1f7-4a28-9410-4b61288dc670"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:28:15.033645 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:15.033608    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtjvh\" (UniqueName: \"kubernetes.io/projected/a7691dde-c1f7-4a28-9410-4b61288dc670-kube-api-access-dtjvh\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:28:15.033731 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:15.033647    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7691dde-c1f7-4a28-9410-4b61288dc670-model-cache\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:28:15.697302 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:15.697268    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mwmwq" event={"ID":"a7691dde-c1f7-4a28-9410-4b61288dc670","Type":"ContainerDied","Data":"c2e67d93be39c82d4957e9e3866769990b8dbbc2eded584ff9c6d4b160e888eb"}
Sep 08 13:28:15.697302 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:15.697307    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2e67d93be39c82d4957e9e3866769990b8dbbc2eded584ff9c6d4b160e888eb"
Sep 08 13:28:15.697509 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:15.697318    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mwmwq"
Sep 08 13:28:28.008296 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.008257    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"]
Sep 08 13:28:28.008882 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.008777    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7691dde-c1f7-4a28-9410-4b61288dc670" containerName="download-hf-model"
Sep 08 13:28:28.008882 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.008791    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7691dde-c1f7-4a28-9410-4b61288dc670" containerName="download-hf-model"
Sep 08 13:28:28.008882 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.008808    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7691dde-c1f7-4a28-9410-4b61288dc670" containerName="s3-init"
Sep 08 13:28:28.008882 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.008814    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7691dde-c1f7-4a28-9410-4b61288dc670" containerName="s3-init"
Sep 08 13:28:28.009086 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.008920    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7691dde-c1f7-4a28-9410-4b61288dc670" containerName="s3-init"
Sep 08 13:28:28.013259 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.013238    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.017175 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.016973    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-e2eab-predictor-serving-cert\""
Sep 08 13:28:28.017175 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.017078    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:28:28.017955 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.017931    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:28:28.018341 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.018319    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-48hwc\""
Sep 08 13:28:28.018448 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.018345    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:28:28.018448 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.018390    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-e2eab-kube-rbac-proxy-sar-config\""
Sep 08 13:28:28.018448 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.018345    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:28:28.019822 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.019802    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"]
Sep 08 13:28:28.041543 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.041516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls8r8\" (UniqueName: \"kubernetes.io/projected/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kube-api-access-ls8r8\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.041695 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.041550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.041695 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.041577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.041695 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.041669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-proxy-tls\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.041866 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.041720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.106534 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.106503    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"]
Sep 08 13:28:28.110088 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.110071    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.113548 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.113527    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 08 13:28:28.113698 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.113565    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 08 13:28:28.129463 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.129437    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"]
Sep 08 13:28:28.142559 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.142535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.142722 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.142576    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/396f8b43-c186-46cd-b863-a8717333c259-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.142722 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.142612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.142722 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.142657    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-proxy-tls\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.142722 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.142692    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/396f8b43-c186-46cd-b863-a8717333c259-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.142930 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.142725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67cnd\" (UniqueName: \"kubernetes.io/projected/396f8b43-c186-46cd-b863-a8717333c259-kube-api-access-67cnd\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.142930 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.142798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.142930 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:28:28.142806    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-e2eab-predictor-serving-cert" not found
Sep 08 13:28:28.142930 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.142865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396f8b43-c186-46cd-b863-a8717333c259-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.142930 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:28:28.142895    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-proxy-tls podName:dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd nodeName:}" failed. No retries permitted until 2026-09-08 13:28:28.642874242 +0000 UTC m=+726.242971861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-proxy-tls") pod "isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" (UID: "dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd") : secret "isvc-raw-sklearn-batcher-e2eab-predictor-serving-cert" not found
Sep 08 13:28:28.143179 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.142989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/396f8b43-c186-46cd-b863-a8717333c259-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.143179 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.143041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls8r8\" (UniqueName: \"kubernetes.io/projected/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kube-api-access-ls8r8\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.143327 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.143303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.143410 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.143392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.143555 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.143541    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.155904 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.155875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls8r8\" (UniqueName: \"kubernetes.io/projected/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kube-api-access-ls8r8\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.196569 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.196541    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"]
Sep 08 13:28:28.200226 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.200208    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.203823 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.203800    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 08 13:28:28.213428 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.213405    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"]
Sep 08 13:28:28.243570 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.243544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/396f8b43-c186-46cd-b863-a8717333c259-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.243728 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.243581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-7584f\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.243728 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.243629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/396f8b43-c186-46cd-b863-a8717333c259-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.243728 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.243675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67cnd\" (UniqueName: \"kubernetes.io/projected/396f8b43-c186-46cd-b863-a8717333c259-kube-api-access-67cnd\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.243728 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.243710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzmct\" (UniqueName: \"kubernetes.io/projected/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kube-api-access-gzmct\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-7584f\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.243956 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.243737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-7584f\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.243956 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.243793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-7584f\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.243956 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.243867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-7584f\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.243956 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.243912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396f8b43-c186-46cd-b863-a8717333c259-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.243956 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.243942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/396f8b43-c186-46cd-b863-a8717333c259-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.244214 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:28:28.244009    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 08 13:28:28.244214 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:28:28.244077    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/396f8b43-c186-46cd-b863-a8717333c259-proxy-tls podName:396f8b43-c186-46cd-b863-a8717333c259 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:28.744061001 +0000 UTC m=+726.344158622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/396f8b43-c186-46cd-b863-a8717333c259-proxy-tls") pod "isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" (UID: "396f8b43-c186-46cd-b863-a8717333c259") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 08 13:28:28.244330 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.244275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/396f8b43-c186-46cd-b863-a8717333c259-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.244330 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.244282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/396f8b43-c186-46cd-b863-a8717333c259-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.244330 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.244289    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/396f8b43-c186-46cd-b863-a8717333c259-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.260204 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.260144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67cnd\" (UniqueName: \"kubernetes.io/projected/396f8b43-c186-46cd-b863-a8717333c259-kube-api-access-67cnd\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.344478 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.344439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-7584f\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.344704 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.344511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-7584f\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.344704 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.344544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzmct\" (UniqueName: \"kubernetes.io/projected/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kube-api-access-gzmct\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-7584f\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.344704 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.344559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-7584f\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.344704 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.344579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-7584f\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.344934 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.344862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-7584f\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.345209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.345188    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-7584f\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.345280 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.345263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-7584f\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.346950 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.346932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-7584f\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.353437 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.353413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzmct\" (UniqueName: \"kubernetes.io/projected/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kube-api-access-gzmct\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-7584f\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.512220 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.512121    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:28.646688 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.646656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-proxy-tls\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.648920 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.648901    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-proxy-tls\") pod \"isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:28.747601 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.747567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396f8b43-c186-46cd-b863-a8717333c259-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.749924 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.749896    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396f8b43-c186-46cd-b863-a8717333c259-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-665748cc96-wdcfm\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:28.785301 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.785233    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"]
Sep 08 13:28:28.788572 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:28:28.788547    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b580a0e_ea94_4c7c_9f4d_b1464e7dbaff.slice/crio-8ff0edf0bd18aa1a99928f78bf74e417075b0e8ca5f999542e5d308dc2e69091 WatchSource:0}: Error finding container 8ff0edf0bd18aa1a99928f78bf74e417075b0e8ca5f999542e5d308dc2e69091: Status 404 returned error can't find the container with id 8ff0edf0bd18aa1a99928f78bf74e417075b0e8ca5f999542e5d308dc2e69091
Sep 08 13:28:28.925638 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:28.925591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:29.021387 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:29.021353    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:29.200053 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:29.200031    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"]
Sep 08 13:28:29.202111 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:28:29.202087    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe45ca6_4181_4a1d_8f8f_8d1b1bbb82cd.slice/crio-4ef4603ffa655ab245567f2fcb75d72c21a6001ccbef3abf78b643e04ea792d8 WatchSource:0}: Error finding container 4ef4603ffa655ab245567f2fcb75d72c21a6001ccbef3abf78b643e04ea792d8: Status 404 returned error can't find the container with id 4ef4603ffa655ab245567f2fcb75d72c21a6001ccbef3abf78b643e04ea792d8
Sep 08 13:28:29.295714 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:29.295685    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"]
Sep 08 13:28:29.297516 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:28:29.297487    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396f8b43_c186_46cd_b863_a8717333c259.slice/crio-a541e861091c00c830689a46bb0e6baefd1650732f80d15787b136ce5f3b0b1e WatchSource:0}: Error finding container a541e861091c00c830689a46bb0e6baefd1650732f80d15787b136ce5f3b0b1e: Status 404 returned error can't find the container with id a541e861091c00c830689a46bb0e6baefd1650732f80d15787b136ce5f3b0b1e
Sep 08 13:28:29.751633 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:29.751583    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" event={"ID":"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff","Type":"ContainerStarted","Data":"0a5a1f4dfba47e73b6930dc9080b87c95bf5d5ed7ecbe2286b37c61a2e72e2dc"}
Sep 08 13:28:29.751826 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:29.751662    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" event={"ID":"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff","Type":"ContainerStarted","Data":"8ff0edf0bd18aa1a99928f78bf74e417075b0e8ca5f999542e5d308dc2e69091"}
Sep 08 13:28:29.753477 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:29.753449    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" event={"ID":"396f8b43-c186-46cd-b863-a8717333c259","Type":"ContainerStarted","Data":"8245d860b301e9f1bd1edf02ca813e38c633608d2278a759c7fd85c328c67920"}
Sep 08 13:28:29.753603 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:29.753512    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" event={"ID":"396f8b43-c186-46cd-b863-a8717333c259","Type":"ContainerStarted","Data":"a541e861091c00c830689a46bb0e6baefd1650732f80d15787b136ce5f3b0b1e"}
Sep 08 13:28:29.755268 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:29.755243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" event={"ID":"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd","Type":"ContainerStarted","Data":"93a27653b016dd240c2d2fc2085647592b7615d3a129d4af3f29b7c434d3552a"}
Sep 08 13:28:29.755361 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:29.755312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" event={"ID":"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd","Type":"ContainerStarted","Data":"4ef4603ffa655ab245567f2fcb75d72c21a6001ccbef3abf78b643e04ea792d8"}
Sep 08 13:28:33.773791 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:33.773748    2578 generic.go:356] "Generic (PLEG): container finished" podID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerID="0a5a1f4dfba47e73b6930dc9080b87c95bf5d5ed7ecbe2286b37c61a2e72e2dc" exitCode=0
Sep 08 13:28:33.774288 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:33.773818    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" event={"ID":"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff","Type":"ContainerDied","Data":"0a5a1f4dfba47e73b6930dc9080b87c95bf5d5ed7ecbe2286b37c61a2e72e2dc"}
Sep 08 13:28:33.775454 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:33.775432    2578 generic.go:356] "Generic (PLEG): container finished" podID="396f8b43-c186-46cd-b863-a8717333c259" containerID="8245d860b301e9f1bd1edf02ca813e38c633608d2278a759c7fd85c328c67920" exitCode=0
Sep 08 13:28:33.775537 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:33.775501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" event={"ID":"396f8b43-c186-46cd-b863-a8717333c259","Type":"ContainerDied","Data":"8245d860b301e9f1bd1edf02ca813e38c633608d2278a759c7fd85c328c67920"}
Sep 08 13:28:33.777128 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:33.777112    2578 generic.go:356] "Generic (PLEG): container finished" podID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerID="93a27653b016dd240c2d2fc2085647592b7615d3a129d4af3f29b7c434d3552a" exitCode=0
Sep 08 13:28:33.777181 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:33.777138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" event={"ID":"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd","Type":"ContainerDied","Data":"93a27653b016dd240c2d2fc2085647592b7615d3a129d4af3f29b7c434d3552a"}
Sep 08 13:28:51.910598 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:51.910552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" event={"ID":"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff","Type":"ContainerStarted","Data":"7993d4e5276329cc9f16e3a811d8ffa0ad08ea36b9632a2692dca7b25e47efec"}
Sep 08 13:28:51.913821 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:51.913792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" event={"ID":"396f8b43-c186-46cd-b863-a8717333c259","Type":"ContainerStarted","Data":"f82a4c6e47d1807450ed4c03b7b0b0eaf3ed04050e3d893deb4c47b90577c2b3"}
Sep 08 13:28:51.918396 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:51.918368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" event={"ID":"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd","Type":"ContainerStarted","Data":"37638b7642cb70962add53095d1dbf673e5396d19552c0e7764e443f49c32d9d"}
Sep 08 13:28:53.928329 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:53.928293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" event={"ID":"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd","Type":"ContainerStarted","Data":"1bc8373d4f032ce346e15bf351cddc5eadbc98029754bd3aba047d82f4a58d6e"}
Sep 08 13:28:53.930335 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:53.930311    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" event={"ID":"396f8b43-c186-46cd-b863-a8717333c259","Type":"ContainerStarted","Data":"8cd925fe4291be963c45406fb406b991fc3887677303567537f14fe8f1527401"}
Sep 08 13:28:53.930530 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:53.930511    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:53.930665 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:53.930555    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:53.931908 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:53.931883    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:28:53.955283 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:53.955238    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" podStartSLOduration=2.48603969 podStartE2EDuration="25.955225421s" totalImagesPullingTime="19.570624419s" totalInitContainerRuntime="3.898561312s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:53.954679251 +0000 UTC m=+751.554776879" watchObservedRunningTime="2026-09-08 13:28:53.955225421 +0000 UTC m=+751.555323047"
Sep 08 13:28:54.936364 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:54.936327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" event={"ID":"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff","Type":"ContainerStarted","Data":"1de72695bc1bec997b8fcbe8b354f2006969fbdb788a4bfc3ab878ab851b2d58"}
Sep 08 13:28:54.936860 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:54.936486    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:54.936860 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:54.936764    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:28:54.936860 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:54.936810    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:28:54.937938 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:54.937914    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:28:54.958581 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:54.958539    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" podStartSLOduration=2.5144640430000003 podStartE2EDuration="26.958525755s" totalImagesPullingTime="20.511644279s" totalInitContainerRuntime="3.932417433s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:54.956796355 +0000 UTC m=+752.556893994" watchObservedRunningTime="2026-09-08 13:28:54.958525755 +0000 UTC m=+752.558623381"
Sep 08 13:28:55.940354 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:55.940305    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:28:59.940872 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:59.940835    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:28:59.941305 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:59.941278    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:28:59.961381 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:59.961353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" event={"ID":"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd","Type":"ContainerStarted","Data":"8c6bfb632651b03a08d983d19e3b28cf0032f7f8b589f7a311bc076e1acbf3f9"}
Sep 08 13:28:59.961551 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:59.961505    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:59.966032 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:59.966012    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:28:59.983923 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:28:59.983878    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podStartSLOduration=3.7663817440000003 podStartE2EDuration="32.983864782s" totalImagesPullingTime="25.57718371s" totalInitContainerRuntime="3.640299328s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:59.981888337 +0000 UTC m=+757.581985988" watchObservedRunningTime="2026-09-08 13:28:59.983864782 +0000 UTC m=+757.583962443"
Sep 08 13:29:00.944494 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:00.944468    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:29:00.944988 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:00.944961    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:29:00.966364 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:00.966340    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:29:00.966525 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:00.966426    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:29:00.966784 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:00.966759    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 08 13:29:00.967201 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:00.967177    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:29:01.971108 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:01.971054    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 08 13:29:01.971595 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:01.971365    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:29:09.941751 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:09.941710    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:29:10.945087 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:10.945045    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:29:11.971671 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:11.971597    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 08 13:29:11.972246 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:11.972216    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:29:19.941455 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:19.941413    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:29:20.945014 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:20.944974    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:29:21.971320 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:21.971272    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 08 13:29:21.971986 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:21.971947    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:29:29.941900 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:29.941859    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:29:30.945632 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:30.945589    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:29:31.971866 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:31.971806    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 08 13:29:31.972368 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:31.972281    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:29:39.941395 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:39.941356    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:29:40.945639 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:40.945582    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:29:41.971143 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:41.971090    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 08 13:29:41.971647 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:41.971572    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:29:49.941402 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:49.941313    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:29:50.945143 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:50.945102    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:29:51.971866 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:51.971819    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 08 13:29:51.972406 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:51.972371    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:29:59.941364 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:29:59.941316    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:30:00.945404 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:00.945363    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:30:01.971578 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:01.971524    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 08 13:30:01.972124 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:01.972091    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:30:02.008149 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:02.008113    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:30:05.008481 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:05.008436    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 08 13:30:05.008918 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:05.008824    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:30:10.945613 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:10.945586    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:30:12.009390 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:12.009361    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:30:15.013206 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:15.013177    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:30:15.013668 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:15.013276    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:30:17.788970 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:17.788934    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"]
Sep 08 13:30:17.789459 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:17.789226    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kserve-container" containerID="cri-o://f82a4c6e47d1807450ed4c03b7b0b0eaf3ed04050e3d893deb4c47b90577c2b3" gracePeriod=30
Sep 08 13:30:17.789459 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:17.789246    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kube-rbac-proxy" containerID="cri-o://8cd925fe4291be963c45406fb406b991fc3887677303567537f14fe8f1527401" gracePeriod=30
Sep 08 13:30:18.289604 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:18.289565    2578 generic.go:356] "Generic (PLEG): container finished" podID="396f8b43-c186-46cd-b863-a8717333c259" containerID="8cd925fe4291be963c45406fb406b991fc3887677303567537f14fe8f1527401" exitCode=2
Sep 08 13:30:18.289787 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:18.289659    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" event={"ID":"396f8b43-c186-46cd-b863-a8717333c259","Type":"ContainerDied","Data":"8cd925fe4291be963c45406fb406b991fc3887677303567537f14fe8f1527401"}
Sep 08 13:30:19.936814 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:19.936771    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Sep 08 13:30:22.008999 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.008960    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Sep 08 13:30:22.315068 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.315036    2578 generic.go:356] "Generic (PLEG): container finished" podID="396f8b43-c186-46cd-b863-a8717333c259" containerID="f82a4c6e47d1807450ed4c03b7b0b0eaf3ed04050e3d893deb4c47b90577c2b3" exitCode=0
Sep 08 13:30:22.315203 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.315112    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" event={"ID":"396f8b43-c186-46cd-b863-a8717333c259","Type":"ContainerDied","Data":"f82a4c6e47d1807450ed4c03b7b0b0eaf3ed04050e3d893deb4c47b90577c2b3"}
Sep 08 13:30:22.325908 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.325888    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:30:22.489162 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.489131    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/396f8b43-c186-46cd-b863-a8717333c259-kube-rbac-proxy-sar-config\") pod \"396f8b43-c186-46cd-b863-a8717333c259\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") "
Sep 08 13:30:22.489162 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.489163    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396f8b43-c186-46cd-b863-a8717333c259-proxy-tls\") pod \"396f8b43-c186-46cd-b863-a8717333c259\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") "
Sep 08 13:30:22.489364 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.489186    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67cnd\" (UniqueName: \"kubernetes.io/projected/396f8b43-c186-46cd-b863-a8717333c259-kube-api-access-67cnd\") pod \"396f8b43-c186-46cd-b863-a8717333c259\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") "
Sep 08 13:30:22.489364 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.489247    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/396f8b43-c186-46cd-b863-a8717333c259-kserve-provision-location\") pod \"396f8b43-c186-46cd-b863-a8717333c259\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") "
Sep 08 13:30:22.489364 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.489306    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/396f8b43-c186-46cd-b863-a8717333c259-cabundle-cert\") pod \"396f8b43-c186-46cd-b863-a8717333c259\" (UID: \"396f8b43-c186-46cd-b863-a8717333c259\") "
Sep 08 13:30:22.489662 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.489520    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396f8b43-c186-46cd-b863-a8717333c259-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "396f8b43-c186-46cd-b863-a8717333c259" (UID: "396f8b43-c186-46cd-b863-a8717333c259"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:30:22.489765 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.489656    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396f8b43-c186-46cd-b863-a8717333c259-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "396f8b43-c186-46cd-b863-a8717333c259" (UID: "396f8b43-c186-46cd-b863-a8717333c259"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:30:22.489765 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.489703    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396f8b43-c186-46cd-b863-a8717333c259-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "396f8b43-c186-46cd-b863-a8717333c259" (UID: "396f8b43-c186-46cd-b863-a8717333c259"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:30:22.491269 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.491248    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396f8b43-c186-46cd-b863-a8717333c259-kube-api-access-67cnd" (OuterVolumeSpecName: "kube-api-access-67cnd") pod "396f8b43-c186-46cd-b863-a8717333c259" (UID: "396f8b43-c186-46cd-b863-a8717333c259"). InnerVolumeSpecName "kube-api-access-67cnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:30:22.491333 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.491309    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396f8b43-c186-46cd-b863-a8717333c259-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "396f8b43-c186-46cd-b863-a8717333c259" (UID: "396f8b43-c186-46cd-b863-a8717333c259"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:30:22.590157 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.590066    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/396f8b43-c186-46cd-b863-a8717333c259-kserve-provision-location\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:30:22.590157 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.590098    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/396f8b43-c186-46cd-b863-a8717333c259-cabundle-cert\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:30:22.590157 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.590112    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/396f8b43-c186-46cd-b863-a8717333c259-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:30:22.590157 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.590126    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396f8b43-c186-46cd-b863-a8717333c259-proxy-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:30:22.590157 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:22.590139    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67cnd\" (UniqueName: \"kubernetes.io/projected/396f8b43-c186-46cd-b863-a8717333c259-kube-api-access-67cnd\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:30:23.196352 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.196319    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"]
Sep 08 13:30:23.196872 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.196588    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kube-rbac-proxy" containerID="cri-o://1bc8373d4f032ce346e15bf351cddc5eadbc98029754bd3aba047d82f4a58d6e" gracePeriod=30
Sep 08 13:30:23.196872 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.196588    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent" containerID="cri-o://8c6bfb632651b03a08d983d19e3b28cf0032f7f8b589f7a311bc076e1acbf3f9" gracePeriod=30
Sep 08 13:30:23.196872 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.196570    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container" containerID="cri-o://37638b7642cb70962add53095d1dbf673e5396d19552c0e7764e443f49c32d9d" gracePeriod=30
Sep 08 13:30:23.322863 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.322834    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"
Sep 08 13:30:23.323034 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.322836    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm" event={"ID":"396f8b43-c186-46cd-b863-a8717333c259","Type":"ContainerDied","Data":"a541e861091c00c830689a46bb0e6baefd1650732f80d15787b136ce5f3b0b1e"}
Sep 08 13:30:23.323034 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.323007    2578 scope.go:117] "RemoveContainer" containerID="8cd925fe4291be963c45406fb406b991fc3887677303567537f14fe8f1527401"
Sep 08 13:30:23.326213 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.326183    2578 generic.go:356] "Generic (PLEG): container finished" podID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerID="1bc8373d4f032ce346e15bf351cddc5eadbc98029754bd3aba047d82f4a58d6e" exitCode=2
Sep 08 13:30:23.326327 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.326222    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" event={"ID":"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd","Type":"ContainerDied","Data":"1bc8373d4f032ce346e15bf351cddc5eadbc98029754bd3aba047d82f4a58d6e"}
Sep 08 13:30:23.331911 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.331887    2578 scope.go:117] "RemoveContainer" containerID="f82a4c6e47d1807450ed4c03b7b0b0eaf3ed04050e3d893deb4c47b90577c2b3"
Sep 08 13:30:23.333675 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.333652    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"]
Sep 08 13:30:23.334013 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.334000    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="storage-initializer"
Sep 08 13:30:23.334064 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.334017    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="storage-initializer"
Sep 08 13:30:23.334064 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.334041    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kserve-container"
Sep 08 13:30:23.334064 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.334048    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kserve-container"
Sep 08 13:30:23.334064 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.334055    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kube-rbac-proxy"
Sep 08 13:30:23.334064 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.334061    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kube-rbac-proxy"
Sep 08 13:30:23.334209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.334118    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kube-rbac-proxy"
Sep 08 13:30:23.334209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.334126    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="396f8b43-c186-46cd-b863-a8717333c259" containerName="kserve-container"
Sep 08 13:30:23.336404 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.336386    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:23.339361 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:30:23.339329    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"isvc-xgboost-graph-raw-3d858-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-134-224.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-134-224.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-3d858-predictor-serving-cert\"" type="*v1.Secret"
Sep 08 13:30:23.339737 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:30:23.339708    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"isvc-xgboost-graph-raw-3d858-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-134-224.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-134-224.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-3d858-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Sep 08 13:30:23.339865 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.339850    2578 scope.go:117] "RemoveContainer" containerID="8245d860b301e9f1bd1edf02ca813e38c633608d2278a759c7fd85c328c67920"
Sep 08 13:30:23.356878 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.356849    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"]
Sep 08 13:30:23.370953 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.370930    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"]
Sep 08 13:30:23.380653 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.380630    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-665748cc96-wdcfm"]
Sep 08 13:30:23.497579 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.497541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:23.497791 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.497591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80c21bdf-084d-4bd1-8c40-b3609b3334fc-proxy-tls\") pod \"isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:23.497791 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.497718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:23.497791 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.497771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80c21bdf-084d-4bd1-8c40-b3609b3334fc-cabundle-cert\") pod \"isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:23.497967 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.497813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2mjx\" (UniqueName: \"kubernetes.io/projected/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kube-api-access-b2mjx\") pod \"isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:23.599033 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.598945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80c21bdf-084d-4bd1-8c40-b3609b3334fc-cabundle-cert\") pod \"isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:23.599033 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.598985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2mjx\" (UniqueName: \"kubernetes.io/projected/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kube-api-access-b2mjx\") pod \"isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:23.599033 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.599021    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:23.599296 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.599064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80c21bdf-084d-4bd1-8c40-b3609b3334fc-proxy-tls\") pod \"isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:23.599296 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.599131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:23.599458 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.599431    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:23.599719 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.599701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80c21bdf-084d-4bd1-8c40-b3609b3334fc-cabundle-cert\") pod \"isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:23.611930 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:23.611906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2mjx\" (UniqueName: \"kubernetes.io/projected/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kube-api-access-b2mjx\") pod \"isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:24.427296 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:24.427263    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-3d858-kube-rbac-proxy-sar-config\""
Sep 08 13:30:24.429858 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:24.429836    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-3d858-predictor-serving-cert\""
Sep 08 13:30:24.429988 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:24.429835    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:24.441470 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:24.441444    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80c21bdf-084d-4bd1-8c40-b3609b3334fc-proxy-tls\") pod \"isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:24.547849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:24.547821    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:24.825055 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:24.825030    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"]
Sep 08 13:30:24.827765 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:30:24.827737    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c21bdf_084d_4bd1_8c40_b3609b3334fc.slice/crio-e42e30c8049370ef43e8de2d44e428fbb050c050c21a2b660932a92d00aeb748 WatchSource:0}: Error finding container e42e30c8049370ef43e8de2d44e428fbb050c050c21a2b660932a92d00aeb748: Status 404 returned error can't find the container with id e42e30c8049370ef43e8de2d44e428fbb050c050c21a2b660932a92d00aeb748
Sep 08 13:30:24.962589 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:24.962538    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 08 13:30:25.008637 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:25.008580    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 08 13:30:25.009101 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:25.009075    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:30:25.014034 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:25.014012    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="396f8b43-c186-46cd-b863-a8717333c259" path="/var/lib/kubelet/pods/396f8b43-c186-46cd-b863-a8717333c259/volumes"
Sep 08 13:30:25.338842 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:25.338805    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" event={"ID":"80c21bdf-084d-4bd1-8c40-b3609b3334fc","Type":"ContainerStarted","Data":"5dd7676b3dce6539f7a6cf0f0d0f064f3e7c66c2e0d2c03323db73758df227cb"}
Sep 08 13:30:25.339047 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:25.338881    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" event={"ID":"80c21bdf-084d-4bd1-8c40-b3609b3334fc","Type":"ContainerStarted","Data":"e42e30c8049370ef43e8de2d44e428fbb050c050c21a2b660932a92d00aeb748"}
Sep 08 13:30:28.355398 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:28.355362    2578 generic.go:356] "Generic (PLEG): container finished" podID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerID="37638b7642cb70962add53095d1dbf673e5396d19552c0e7764e443f49c32d9d" exitCode=0
Sep 08 13:30:28.355798 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:28.355430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" event={"ID":"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd","Type":"ContainerDied","Data":"37638b7642cb70962add53095d1dbf673e5396d19552c0e7764e443f49c32d9d"}
Sep 08 13:30:29.361170 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:29.361134    2578 generic.go:356] "Generic (PLEG): container finished" podID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerID="5dd7676b3dce6539f7a6cf0f0d0f064f3e7c66c2e0d2c03323db73758df227cb" exitCode=0
Sep 08 13:30:29.361573 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:29.361206    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" event={"ID":"80c21bdf-084d-4bd1-8c40-b3609b3334fc","Type":"ContainerDied","Data":"5dd7676b3dce6539f7a6cf0f0d0f064f3e7c66c2e0d2c03323db73758df227cb"}
Sep 08 13:30:29.962642 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:29.962356    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 08 13:30:34.962352 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:34.962313    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 08 13:30:34.962751 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:34.962424    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:30:35.008932 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:35.008896    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 08 13:30:35.009260 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:35.009230    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:30:39.962914 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:39.962864    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 08 13:30:44.962076 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:44.962037    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 08 13:30:45.008976 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:45.008929    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 08 13:30:45.009376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:45.009333    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:30:45.014001 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:45.013979    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:30:45.014119 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:45.014104    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:30:48.470458 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:48.470423    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" event={"ID":"80c21bdf-084d-4bd1-8c40-b3609b3334fc","Type":"ContainerStarted","Data":"0e96481cc1401b2433294cb3c5971bd0a193ce2cdebec6b2e14a9a12fccaea37"}
Sep 08 13:30:48.470960 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:48.470485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" event={"ID":"80c21bdf-084d-4bd1-8c40-b3609b3334fc","Type":"ContainerStarted","Data":"6591fe2896b184523457cedd1b278a1432e8dbfd1a463b8b150becb3f70a5441"}
Sep 08 13:30:48.470960 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:48.470676    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:48.500075 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:48.496799    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" podStartSLOduration=3.36599931 podStartE2EDuration="25.496780227s" totalImagesPullingTime="18.55224156s" totalInitContainerRuntime="3.578539357s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:30:48.494904572 +0000 UTC m=+866.095002199" watchObservedRunningTime="2026-09-08 13:30:48.496780227 +0000 UTC m=+866.096877854"
Sep 08 13:30:49.475317 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:49.475282    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:49.476673 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:49.476638    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:30:49.961887 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:49.961829    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 08 13:30:50.479707 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:50.479664    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:30:53.342605 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.342583    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:30:53.467990 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.467913    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls8r8\" (UniqueName: \"kubernetes.io/projected/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kube-api-access-ls8r8\") pod \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") "
Sep 08 13:30:53.467990 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.467988    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-cabundle-cert\") pod \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") "
Sep 08 13:30:53.468217 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.468022    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-proxy-tls\") pod \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") "
Sep 08 13:30:53.468217 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.468068    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kserve-provision-location\") pod \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") "
Sep 08 13:30:53.468217 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.468089    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kube-rbac-proxy-sar-config\") pod \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\" (UID: \"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd\") "
Sep 08 13:30:53.468455 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.468366    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" (UID: "dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:30:53.468540 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.468455    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" (UID: "dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:30:53.468540 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.468514    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" (UID: "dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:30:53.470166 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.470141    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kube-api-access-ls8r8" (OuterVolumeSpecName: "kube-api-access-ls8r8") pod "dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" (UID: "dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd"). InnerVolumeSpecName "kube-api-access-ls8r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:30:53.470243 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.470198    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" (UID: "dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:30:53.493262 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.493232    2578 generic.go:356] "Generic (PLEG): container finished" podID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerID="8c6bfb632651b03a08d983d19e3b28cf0032f7f8b589f7a311bc076e1acbf3f9" exitCode=0
Sep 08 13:30:53.493366 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.493301    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"
Sep 08 13:30:53.493366 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.493315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" event={"ID":"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd","Type":"ContainerDied","Data":"8c6bfb632651b03a08d983d19e3b28cf0032f7f8b589f7a311bc076e1acbf3f9"}
Sep 08 13:30:53.493499 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.493383    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp" event={"ID":"dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd","Type":"ContainerDied","Data":"4ef4603ffa655ab245567f2fcb75d72c21a6001ccbef3abf78b643e04ea792d8"}
Sep 08 13:30:53.493499 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.493416    2578 scope.go:117] "RemoveContainer" containerID="8c6bfb632651b03a08d983d19e3b28cf0032f7f8b589f7a311bc076e1acbf3f9"
Sep 08 13:30:53.503058 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.503031    2578 scope.go:117] "RemoveContainer" containerID="1bc8373d4f032ce346e15bf351cddc5eadbc98029754bd3aba047d82f4a58d6e"
Sep 08 13:30:53.510939 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.510921    2578 scope.go:117] "RemoveContainer" containerID="37638b7642cb70962add53095d1dbf673e5396d19552c0e7764e443f49c32d9d"
Sep 08 13:30:53.518181 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.518160    2578 scope.go:117] "RemoveContainer" containerID="93a27653b016dd240c2d2fc2085647592b7615d3a129d4af3f29b7c434d3552a"
Sep 08 13:30:53.525694 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.525670    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"]
Sep 08 13:30:53.526325 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.526301    2578 scope.go:117] "RemoveContainer" containerID="8c6bfb632651b03a08d983d19e3b28cf0032f7f8b589f7a311bc076e1acbf3f9"
Sep 08 13:30:53.526864 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:30:53.526818    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6bfb632651b03a08d983d19e3b28cf0032f7f8b589f7a311bc076e1acbf3f9\": container with ID starting with 8c6bfb632651b03a08d983d19e3b28cf0032f7f8b589f7a311bc076e1acbf3f9 not found: ID does not exist" containerID="8c6bfb632651b03a08d983d19e3b28cf0032f7f8b589f7a311bc076e1acbf3f9"
Sep 08 13:30:53.526983 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.526872    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6bfb632651b03a08d983d19e3b28cf0032f7f8b589f7a311bc076e1acbf3f9"} err="failed to get container status \"8c6bfb632651b03a08d983d19e3b28cf0032f7f8b589f7a311bc076e1acbf3f9\": rpc error: code = NotFound desc = could not find container \"8c6bfb632651b03a08d983d19e3b28cf0032f7f8b589f7a311bc076e1acbf3f9\": container with ID starting with 8c6bfb632651b03a08d983d19e3b28cf0032f7f8b589f7a311bc076e1acbf3f9 not found: ID does not exist"
Sep 08 13:30:53.526983 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.526899    2578 scope.go:117] "RemoveContainer" containerID="1bc8373d4f032ce346e15bf351cddc5eadbc98029754bd3aba047d82f4a58d6e"
Sep 08 13:30:53.527236 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:30:53.527218    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc8373d4f032ce346e15bf351cddc5eadbc98029754bd3aba047d82f4a58d6e\": container with ID starting with 1bc8373d4f032ce346e15bf351cddc5eadbc98029754bd3aba047d82f4a58d6e not found: ID does not exist" containerID="1bc8373d4f032ce346e15bf351cddc5eadbc98029754bd3aba047d82f4a58d6e"
Sep 08 13:30:53.527302 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.527246    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc8373d4f032ce346e15bf351cddc5eadbc98029754bd3aba047d82f4a58d6e"} err="failed to get container status \"1bc8373d4f032ce346e15bf351cddc5eadbc98029754bd3aba047d82f4a58d6e\": rpc error: code = NotFound desc = could not find container \"1bc8373d4f032ce346e15bf351cddc5eadbc98029754bd3aba047d82f4a58d6e\": container with ID starting with 1bc8373d4f032ce346e15bf351cddc5eadbc98029754bd3aba047d82f4a58d6e not found: ID does not exist"
Sep 08 13:30:53.527302 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.527268    2578 scope.go:117] "RemoveContainer" containerID="37638b7642cb70962add53095d1dbf673e5396d19552c0e7764e443f49c32d9d"
Sep 08 13:30:53.527533 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:30:53.527512    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37638b7642cb70962add53095d1dbf673e5396d19552c0e7764e443f49c32d9d\": container with ID starting with 37638b7642cb70962add53095d1dbf673e5396d19552c0e7764e443f49c32d9d not found: ID does not exist" containerID="37638b7642cb70962add53095d1dbf673e5396d19552c0e7764e443f49c32d9d"
Sep 08 13:30:53.527606 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.527543    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37638b7642cb70962add53095d1dbf673e5396d19552c0e7764e443f49c32d9d"} err="failed to get container status \"37638b7642cb70962add53095d1dbf673e5396d19552c0e7764e443f49c32d9d\": rpc error: code = NotFound desc = could not find container \"37638b7642cb70962add53095d1dbf673e5396d19552c0e7764e443f49c32d9d\": container with ID starting with 37638b7642cb70962add53095d1dbf673e5396d19552c0e7764e443f49c32d9d not found: ID does not exist"
Sep 08 13:30:53.527606 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.527576    2578 scope.go:117] "RemoveContainer" containerID="93a27653b016dd240c2d2fc2085647592b7615d3a129d4af3f29b7c434d3552a"
Sep 08 13:30:53.527837 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:30:53.527819    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93a27653b016dd240c2d2fc2085647592b7615d3a129d4af3f29b7c434d3552a\": container with ID starting with 93a27653b016dd240c2d2fc2085647592b7615d3a129d4af3f29b7c434d3552a not found: ID does not exist" containerID="93a27653b016dd240c2d2fc2085647592b7615d3a129d4af3f29b7c434d3552a"
Sep 08 13:30:53.527886 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.527846    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93a27653b016dd240c2d2fc2085647592b7615d3a129d4af3f29b7c434d3552a"} err="failed to get container status \"93a27653b016dd240c2d2fc2085647592b7615d3a129d4af3f29b7c434d3552a\": rpc error: code = NotFound desc = could not find container \"93a27653b016dd240c2d2fc2085647592b7615d3a129d4af3f29b7c434d3552a\": container with ID starting with 93a27653b016dd240c2d2fc2085647592b7615d3a129d4af3f29b7c434d3552a not found: ID does not exist"
Sep 08 13:30:53.528462 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.528445    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-e2eab-predictor-5b888c4b6d-7zcxp"]
Sep 08 13:30:53.569431 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.569407    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls8r8\" (UniqueName: \"kubernetes.io/projected/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kube-api-access-ls8r8\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:30:53.569431 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.569429    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-cabundle-cert\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:30:53.569574 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.569441    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-proxy-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:30:53.569574 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.569449    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kserve-provision-location\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:30:53.569574 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:53.569457    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:30:55.013778 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:55.013745    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" path="/var/lib/kubelet/pods/dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd/volumes"
Sep 08 13:30:55.484371 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:55.484343    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:30:55.484831 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:30:55.484801    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:31:05.485385 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:05.485337    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:31:15.485082 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:15.484996    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:31:25.485102 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:25.485062    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:31:35.485000 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:35.484961    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:31:45.484810 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:45.484770    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:31:47.950499 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:47.950464    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"]
Sep 08 13:31:47.950975 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:47.950762    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kserve-container" containerID="cri-o://7993d4e5276329cc9f16e3a811d8ffa0ad08ea36b9632a2692dca7b25e47efec" gracePeriod=30
Sep 08 13:31:47.950975 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:47.950806    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kube-rbac-proxy" containerID="cri-o://1de72695bc1bec997b8fcbe8b354f2006969fbdb788a4bfc3ab878ab851b2d58" gracePeriod=30
Sep 08 13:31:48.737475 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:48.737440    2578 generic.go:356] "Generic (PLEG): container finished" podID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerID="1de72695bc1bec997b8fcbe8b354f2006969fbdb788a4bfc3ab878ab851b2d58" exitCode=2
Sep 08 13:31:48.737665 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:48.737502    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" event={"ID":"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff","Type":"ContainerDied","Data":"1de72695bc1bec997b8fcbe8b354f2006969fbdb788a4bfc3ab878ab851b2d58"}
Sep 08 13:31:50.940937 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:50.940891    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 08 13:31:50.945256 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:50.945226    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 08 13:31:52.581757 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.581733    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:31:52.644822 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.644797    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-cabundle-cert\") pod \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") "
Sep 08 13:31:52.644982 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.644883    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzmct\" (UniqueName: \"kubernetes.io/projected/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kube-api-access-gzmct\") pod \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") "
Sep 08 13:31:52.644982 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.644927    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kube-rbac-proxy-sar-config\") pod \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") "
Sep 08 13:31:52.644982 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.644959    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kserve-provision-location\") pod \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") "
Sep 08 13:31:52.645142 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.644990    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-proxy-tls\") pod \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\" (UID: \"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff\") "
Sep 08 13:31:52.645287 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.645157    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" (UID: "6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:31:52.645410 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.645284    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" (UID: "6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:31:52.645410 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.645347    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" (UID: "6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:31:52.646892 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.646861    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kube-api-access-gzmct" (OuterVolumeSpecName: "kube-api-access-gzmct") pod "6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" (UID: "6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff"). InnerVolumeSpecName "kube-api-access-gzmct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:31:52.647032 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.647010    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" (UID: "6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:31:52.746412 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.746374    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzmct\" (UniqueName: \"kubernetes.io/projected/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kube-api-access-gzmct\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:31:52.746412 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.746408    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:31:52.746412 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.746418    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-kserve-provision-location\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:31:52.746598 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.746427    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-proxy-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:31:52.746598 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.746435    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff-cabundle-cert\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:31:52.755467 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.755439    2578 generic.go:356] "Generic (PLEG): container finished" podID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerID="7993d4e5276329cc9f16e3a811d8ffa0ad08ea36b9632a2692dca7b25e47efec" exitCode=0
Sep 08 13:31:52.755563 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.755506    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"
Sep 08 13:31:52.755563 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.755527    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" event={"ID":"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff","Type":"ContainerDied","Data":"7993d4e5276329cc9f16e3a811d8ffa0ad08ea36b9632a2692dca7b25e47efec"}
Sep 08 13:31:52.755648 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.755587    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f" event={"ID":"6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff","Type":"ContainerDied","Data":"8ff0edf0bd18aa1a99928f78bf74e417075b0e8ca5f999542e5d308dc2e69091"}
Sep 08 13:31:52.755648 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.755613    2578 scope.go:117] "RemoveContainer" containerID="1de72695bc1bec997b8fcbe8b354f2006969fbdb788a4bfc3ab878ab851b2d58"
Sep 08 13:31:52.764093 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.764074    2578 scope.go:117] "RemoveContainer" containerID="7993d4e5276329cc9f16e3a811d8ffa0ad08ea36b9632a2692dca7b25e47efec"
Sep 08 13:31:52.771276 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.771259    2578 scope.go:117] "RemoveContainer" containerID="0a5a1f4dfba47e73b6930dc9080b87c95bf5d5ed7ecbe2286b37c61a2e72e2dc"
Sep 08 13:31:52.778031 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.778010    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"]
Sep 08 13:31:52.779354 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.779317    2578 scope.go:117] "RemoveContainer" containerID="1de72695bc1bec997b8fcbe8b354f2006969fbdb788a4bfc3ab878ab851b2d58"
Sep 08 13:31:52.780055 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:31:52.780034    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de72695bc1bec997b8fcbe8b354f2006969fbdb788a4bfc3ab878ab851b2d58\": container with ID starting with 1de72695bc1bec997b8fcbe8b354f2006969fbdb788a4bfc3ab878ab851b2d58 not found: ID does not exist" containerID="1de72695bc1bec997b8fcbe8b354f2006969fbdb788a4bfc3ab878ab851b2d58"
Sep 08 13:31:52.780132 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.780062    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de72695bc1bec997b8fcbe8b354f2006969fbdb788a4bfc3ab878ab851b2d58"} err="failed to get container status \"1de72695bc1bec997b8fcbe8b354f2006969fbdb788a4bfc3ab878ab851b2d58\": rpc error: code = NotFound desc = could not find container \"1de72695bc1bec997b8fcbe8b354f2006969fbdb788a4bfc3ab878ab851b2d58\": container with ID starting with 1de72695bc1bec997b8fcbe8b354f2006969fbdb788a4bfc3ab878ab851b2d58 not found: ID does not exist"
Sep 08 13:31:52.780132 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.780079    2578 scope.go:117] "RemoveContainer" containerID="7993d4e5276329cc9f16e3a811d8ffa0ad08ea36b9632a2692dca7b25e47efec"
Sep 08 13:31:52.780364 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:31:52.780344    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7993d4e5276329cc9f16e3a811d8ffa0ad08ea36b9632a2692dca7b25e47efec\": container with ID starting with 7993d4e5276329cc9f16e3a811d8ffa0ad08ea36b9632a2692dca7b25e47efec not found: ID does not exist" containerID="7993d4e5276329cc9f16e3a811d8ffa0ad08ea36b9632a2692dca7b25e47efec"
Sep 08 13:31:52.780432 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.780372    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7993d4e5276329cc9f16e3a811d8ffa0ad08ea36b9632a2692dca7b25e47efec"} err="failed to get container status \"7993d4e5276329cc9f16e3a811d8ffa0ad08ea36b9632a2692dca7b25e47efec\": rpc error: code = NotFound desc = could not find container \"7993d4e5276329cc9f16e3a811d8ffa0ad08ea36b9632a2692dca7b25e47efec\": container with ID starting with 7993d4e5276329cc9f16e3a811d8ffa0ad08ea36b9632a2692dca7b25e47efec not found: ID does not exist"
Sep 08 13:31:52.780432 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.780394    2578 scope.go:117] "RemoveContainer" containerID="0a5a1f4dfba47e73b6930dc9080b87c95bf5d5ed7ecbe2286b37c61a2e72e2dc"
Sep 08 13:31:52.780793 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:31:52.780769    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a5a1f4dfba47e73b6930dc9080b87c95bf5d5ed7ecbe2286b37c61a2e72e2dc\": container with ID starting with 0a5a1f4dfba47e73b6930dc9080b87c95bf5d5ed7ecbe2286b37c61a2e72e2dc not found: ID does not exist" containerID="0a5a1f4dfba47e73b6930dc9080b87c95bf5d5ed7ecbe2286b37c61a2e72e2dc"
Sep 08 13:31:52.780867 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.780804    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a5a1f4dfba47e73b6930dc9080b87c95bf5d5ed7ecbe2286b37c61a2e72e2dc"} err="failed to get container status \"0a5a1f4dfba47e73b6930dc9080b87c95bf5d5ed7ecbe2286b37c61a2e72e2dc\": rpc error: code = NotFound desc = could not find container \"0a5a1f4dfba47e73b6930dc9080b87c95bf5d5ed7ecbe2286b37c61a2e72e2dc\": container with ID starting with 0a5a1f4dfba47e73b6930dc9080b87c95bf5d5ed7ecbe2286b37c61a2e72e2dc not found: ID does not exist"
Sep 08 13:31:52.781228 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:52.781212    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-7584f"]
Sep 08 13:31:53.012927 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.012842    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" path="/var/lib/kubelet/pods/6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff/volumes"
Sep 08 13:31:53.145037 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145004    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"]
Sep 08 13:31:53.145466 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145451    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container"
Sep 08 13:31:53.145528 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145468    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container"
Sep 08 13:31:53.145528 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145491    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="storage-initializer"
Sep 08 13:31:53.145528 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145499    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="storage-initializer"
Sep 08 13:31:53.145528 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145513    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent"
Sep 08 13:31:53.145528 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145522    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent"
Sep 08 13:31:53.145800 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145535    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kube-rbac-proxy"
Sep 08 13:31:53.145800 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145543    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kube-rbac-proxy"
Sep 08 13:31:53.145800 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145557    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kserve-container"
Sep 08 13:31:53.145800 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145564    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kserve-container"
Sep 08 13:31:53.145800 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145580    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kube-rbac-proxy"
Sep 08 13:31:53.145800 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145589    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kube-rbac-proxy"
Sep 08 13:31:53.145800 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145606    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="storage-initializer"
Sep 08 13:31:53.145800 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145614    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="storage-initializer"
Sep 08 13:31:53.145800 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145694    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="agent"
Sep 08 13:31:53.145800 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145707    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kube-rbac-proxy"
Sep 08 13:31:53.145800 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145716    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kube-rbac-proxy"
Sep 08 13:31:53.145800 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145725    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbe45ca6-4181-4a1d-8f8f-8d1b1bbb82cd" containerName="kserve-container"
Sep 08 13:31:53.145800 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.145733    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b580a0e-ea94-4c7c-9f4d-b1464e7dbaff" containerName="kserve-container"
Sep 08 13:31:53.148535 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.148517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.151186 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.151161    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 08 13:31:53.151317 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.151161    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 08 13:31:53.158199 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.158178    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"]
Sep 08 13:31:53.218134 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.218097    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"]
Sep 08 13:31:53.220514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.220498    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.223105 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.223078    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 08 13:31:53.231779 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.231757    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"]
Sep 08 13:31:53.250181 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.250154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.250309 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.250196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.250309 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.250273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.250417 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.250352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.250417 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.250385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l42qc\" (UniqueName: \"kubernetes.io/projected/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kube-api-access-l42qc\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.250503 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.250415    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbdc6\" (UniqueName: \"kubernetes.io/projected/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kube-api-access-rbdc6\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.250503 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.250440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.250576 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.250503    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22262be0-d3bc-4a39-b2a5-b0ae2d124092-proxy-tls\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.250576 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.250554    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22262be0-d3bc-4a39-b2a5-b0ae2d124092-cabundle-cert\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.250683 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.250592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.351738 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.351650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.351738 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.351703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.351958 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.351754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.351958 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.351796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.351958 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.351823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l42qc\" (UniqueName: \"kubernetes.io/projected/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kube-api-access-l42qc\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.351958 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.351849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbdc6\" (UniqueName: \"kubernetes.io/projected/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kube-api-access-rbdc6\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.351958 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.351874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.351958 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.351903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22262be0-d3bc-4a39-b2a5-b0ae2d124092-proxy-tls\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.351958 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.351945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22262be0-d3bc-4a39-b2a5-b0ae2d124092-cabundle-cert\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.352341 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.351986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.352341 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.352114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.352341 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:31:53.352206    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 08 13:31:53.352341 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:31:53.352269    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-proxy-tls podName:eb649e86-0ada-48fb-8e65-d9ee1504b5d0 nodeName:}" failed. No retries permitted until 2026-09-08 13:31:53.852249428 +0000 UTC m=+931.452347045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-proxy-tls") pod "isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" (UID: "eb649e86-0ada-48fb-8e65-d9ee1504b5d0") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 08 13:31:53.352610 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.352491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.352610 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:31:53.352534    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 08 13:31:53.352610 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.352540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.352610 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:31:53.352587    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22262be0-d3bc-4a39-b2a5-b0ae2d124092-proxy-tls podName:22262be0-d3bc-4a39-b2a5-b0ae2d124092 nodeName:}" failed. No retries permitted until 2026-09-08 13:31:53.8525721 +0000 UTC m=+931.452669706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/22262be0-d3bc-4a39-b2a5-b0ae2d124092-proxy-tls") pod "isvc-canary-stop-predictor-7b7754b5bf-q8ms2" (UID: "22262be0-d3bc-4a39-b2a5-b0ae2d124092") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 08 13:31:53.352610 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.352604    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.352839 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.352649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.353080 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.353061    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22262be0-d3bc-4a39-b2a5-b0ae2d124092-cabundle-cert\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.362909 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.362881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbdc6\" (UniqueName: \"kubernetes.io/projected/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kube-api-access-rbdc6\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.363046 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.362959    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l42qc\" (UniqueName: \"kubernetes.io/projected/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kube-api-access-l42qc\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.857981 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.857940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:53.858445 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.858019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22262be0-d3bc-4a39-b2a5-b0ae2d124092-proxy-tls\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.861160 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.861126    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22262be0-d3bc-4a39-b2a5-b0ae2d124092-proxy-tls\") pod \"isvc-canary-stop-predictor-7b7754b5bf-q8ms2\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:53.861483 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:53.861459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:54.062474 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:54.062420    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:54.131389 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:54.131353    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:54.335531 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:54.335505    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"]
Sep 08 13:31:54.337099 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:31:54.337071    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb649e86_0ada_48fb_8e65_d9ee1504b5d0.slice/crio-d18a4cb93d72f08aab1dd30d6ae2587cd1ee66efc478c5133a7d971c1adcd459 WatchSource:0}: Error finding container d18a4cb93d72f08aab1dd30d6ae2587cd1ee66efc478c5133a7d971c1adcd459: Status 404 returned error can't find the container with id d18a4cb93d72f08aab1dd30d6ae2587cd1ee66efc478c5133a7d971c1adcd459
Sep 08 13:31:54.402044 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:54.402019    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"]
Sep 08 13:31:54.403718 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:31:54.403691    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22262be0_d3bc_4a39_b2a5_b0ae2d124092.slice/crio-03105da687db624d2248da0ada7199323dcea938fdf00f3b6bb77a74e4add611 WatchSource:0}: Error finding container 03105da687db624d2248da0ada7199323dcea938fdf00f3b6bb77a74e4add611: Status 404 returned error can't find the container with id 03105da687db624d2248da0ada7199323dcea938fdf00f3b6bb77a74e4add611
Sep 08 13:31:54.766151 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:54.766109    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" event={"ID":"eb649e86-0ada-48fb-8e65-d9ee1504b5d0","Type":"ContainerStarted","Data":"de179efe4e6482618cda131bbb87eb3fe9b03aa272913d453e06e3c58fd5b4cb"}
Sep 08 13:31:54.766347 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:54.766193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" event={"ID":"eb649e86-0ada-48fb-8e65-d9ee1504b5d0","Type":"ContainerStarted","Data":"d18a4cb93d72f08aab1dd30d6ae2587cd1ee66efc478c5133a7d971c1adcd459"}
Sep 08 13:31:54.767768 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:54.767741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" event={"ID":"22262be0-d3bc-4a39-b2a5-b0ae2d124092","Type":"ContainerStarted","Data":"cfae7d2532149f42c7453a706487387fa5a1d0a409430b8c38b4db5f121f8b6a"}
Sep 08 13:31:54.767878 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:54.767797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" event={"ID":"22262be0-d3bc-4a39-b2a5-b0ae2d124092","Type":"ContainerStarted","Data":"03105da687db624d2248da0ada7199323dcea938fdf00f3b6bb77a74e4add611"}
Sep 08 13:31:55.485796 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:55.485768    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:31:58.785670 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:58.785608    2578 generic.go:356] "Generic (PLEG): container finished" podID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerID="cfae7d2532149f42c7453a706487387fa5a1d0a409430b8c38b4db5f121f8b6a" exitCode=0
Sep 08 13:31:58.786165 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:58.785681    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" event={"ID":"22262be0-d3bc-4a39-b2a5-b0ae2d124092","Type":"ContainerDied","Data":"cfae7d2532149f42c7453a706487387fa5a1d0a409430b8c38b4db5f121f8b6a"}
Sep 08 13:31:58.787459 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:58.787411    2578 generic.go:356] "Generic (PLEG): container finished" podID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerID="de179efe4e6482618cda131bbb87eb3fe9b03aa272913d453e06e3c58fd5b4cb" exitCode=0
Sep 08 13:31:58.787508 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:58.787473    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" event={"ID":"eb649e86-0ada-48fb-8e65-d9ee1504b5d0","Type":"ContainerDied","Data":"de179efe4e6482618cda131bbb87eb3fe9b03aa272913d453e06e3c58fd5b4cb"}
Sep 08 13:31:59.794823 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:59.794780    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" event={"ID":"22262be0-d3bc-4a39-b2a5-b0ae2d124092","Type":"ContainerStarted","Data":"481a9f90c183deb8f77855dfc5fccc0025ed4030a884a017162b81bf46de486c"}
Sep 08 13:31:59.795283 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:59.794894    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" event={"ID":"22262be0-d3bc-4a39-b2a5-b0ae2d124092","Type":"ContainerStarted","Data":"dafdfb4f7cbeb64513bd25c7533a38ef4f16cf900794e4aa0c52b55751723196"}
Sep 08 13:31:59.795283 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:59.795095    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:31:59.797081 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:59.797056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" event={"ID":"eb649e86-0ada-48fb-8e65-d9ee1504b5d0","Type":"ContainerStarted","Data":"fbe703a741b4ed119f4f51486100a328e7e91160ed19a4c3c8ce647d912735f6"}
Sep 08 13:31:59.797197 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:59.797106    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" event={"ID":"eb649e86-0ada-48fb-8e65-d9ee1504b5d0","Type":"ContainerStarted","Data":"bca0c27764f77908cd93f8157f5344aa9c08fb528f309022cff101b117ee9b4a"}
Sep 08 13:31:59.797361 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:59.797344    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:59.797428 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:59.797393    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:31:59.798764 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:59.798738    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:31:59.820049 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:59.820004    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" podStartSLOduration=3.021787278 podStartE2EDuration="6.819991622s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.798204344s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:31:59.817018378 +0000 UTC m=+937.417116010" watchObservedRunningTime="2026-09-08 13:31:59.819991622 +0000 UTC m=+937.420089248"
Sep 08 13:31:59.838595 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:31:59.838556    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" podStartSLOduration=3.064702856 podStartE2EDuration="6.838544847s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.773841991s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:31:59.837783294 +0000 UTC m=+937.437880918" watchObservedRunningTime="2026-09-08 13:31:59.838544847 +0000 UTC m=+937.438642531"
Sep 08 13:32:00.802953 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:00.802917    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:32:00.802953 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:00.802950    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:32:00.804047 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:00.804024    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:32:01.807178 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:01.807135    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:32:05.807733 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:05.807704    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:32:05.808218 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:05.808190    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:32:06.812057 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:06.812026    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:32:06.812513 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:06.812437    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:32:13.312043 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.311965    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"]
Sep 08 13:32:13.312440 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.312198    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kserve-container" containerID="cri-o://6591fe2896b184523457cedd1b278a1432e8dbfd1a463b8b150becb3f70a5441" gracePeriod=30
Sep 08 13:32:13.312440 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.312235    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kube-rbac-proxy" containerID="cri-o://0e96481cc1401b2433294cb3c5971bd0a193ce2cdebec6b2e14a9a12fccaea37" gracePeriod=30
Sep 08 13:32:13.511825 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.511790    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"]
Sep 08 13:32:13.514847 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.514824    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:13.517994 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.517969    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-90436-predictor-serving-cert\""
Sep 08 13:32:13.518105 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.518004    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-90436-kube-rbac-proxy-sar-config\""
Sep 08 13:32:13.529960 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.529935    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"]
Sep 08 13:32:13.636000 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.635908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34859624-406c-4dae-974c-89b725d34800-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:13.636000 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.635955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34859624-406c-4dae-974c-89b725d34800-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:13.636000 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.635995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34859624-406c-4dae-974c-89b725d34800-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:13.636268 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.636016    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34859624-406c-4dae-974c-89b725d34800-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:13.636268 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.636047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4z9d\" (UniqueName: \"kubernetes.io/projected/34859624-406c-4dae-974c-89b725d34800-kube-api-access-d4z9d\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:13.736689 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.736649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34859624-406c-4dae-974c-89b725d34800-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:13.736862 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.736708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34859624-406c-4dae-974c-89b725d34800-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:13.736862 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.736780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34859624-406c-4dae-974c-89b725d34800-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:13.736862 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.736810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34859624-406c-4dae-974c-89b725d34800-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:13.736862 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.736842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4z9d\" (UniqueName: \"kubernetes.io/projected/34859624-406c-4dae-974c-89b725d34800-kube-api-access-d4z9d\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:13.737127 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:32:13.736984    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-90436-predictor-serving-cert" not found
Sep 08 13:32:13.737127 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:32:13.737059    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34859624-406c-4dae-974c-89b725d34800-proxy-tls podName:34859624-406c-4dae-974c-89b725d34800 nodeName:}" failed. No retries permitted until 2026-09-08 13:32:14.237036093 +0000 UTC m=+951.837133702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/34859624-406c-4dae-974c-89b725d34800-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" (UID: "34859624-406c-4dae-974c-89b725d34800") : secret "isvc-xgboost-graph-raw-hpa-90436-predictor-serving-cert" not found
Sep 08 13:32:13.737204 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.737131    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34859624-406c-4dae-974c-89b725d34800-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:13.737426 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.737408    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34859624-406c-4dae-974c-89b725d34800-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:13.737501 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.737485    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34859624-406c-4dae-974c-89b725d34800-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:13.746409 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.746387    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4z9d\" (UniqueName: \"kubernetes.io/projected/34859624-406c-4dae-974c-89b725d34800-kube-api-access-d4z9d\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:13.858372 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.858336    2578 generic.go:356] "Generic (PLEG): container finished" podID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerID="0e96481cc1401b2433294cb3c5971bd0a193ce2cdebec6b2e14a9a12fccaea37" exitCode=2
Sep 08 13:32:13.858528 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:13.858400    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" event={"ID":"80c21bdf-084d-4bd1-8c40-b3609b3334fc","Type":"ContainerDied","Data":"0e96481cc1401b2433294cb3c5971bd0a193ce2cdebec6b2e14a9a12fccaea37"}
Sep 08 13:32:14.242119 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:14.242081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34859624-406c-4dae-974c-89b725d34800-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:14.244439 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:14.244408    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34859624-406c-4dae-974c-89b725d34800-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:14.424871 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:14.424830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:14.702494 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:14.702469    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"]
Sep 08 13:32:14.705253 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:32:14.705231    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34859624_406c_4dae_974c_89b725d34800.slice/crio-7cc6e48b163ba8980cf51bd8e92c444d24ae368e8e32ff5a095cacf16e7382ce WatchSource:0}: Error finding container 7cc6e48b163ba8980cf51bd8e92c444d24ae368e8e32ff5a095cacf16e7382ce: Status 404 returned error can't find the container with id 7cc6e48b163ba8980cf51bd8e92c444d24ae368e8e32ff5a095cacf16e7382ce
Sep 08 13:32:14.864845 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:14.864811    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" event={"ID":"34859624-406c-4dae-974c-89b725d34800","Type":"ContainerStarted","Data":"cd44002f5b23f866e8c6f5af3e8142e6244bf9a519bb96bcba36eba677446484"}
Sep 08 13:32:14.865032 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:14.864882    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" event={"ID":"34859624-406c-4dae-974c-89b725d34800","Type":"ContainerStarted","Data":"7cc6e48b163ba8980cf51bd8e92c444d24ae368e8e32ff5a095cacf16e7382ce"}
Sep 08 13:32:15.480746 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:15.480698    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 08 13:32:15.485084 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:15.485056    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 08 13:32:15.809032 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:15.808943    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:32:16.813397 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:16.813359    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:32:17.157227 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.157202    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:32:17.266930 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.266897    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2mjx\" (UniqueName: \"kubernetes.io/projected/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kube-api-access-b2mjx\") pod \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") "
Sep 08 13:32:17.267102 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.266945    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80c21bdf-084d-4bd1-8c40-b3609b3334fc-proxy-tls\") pod \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") "
Sep 08 13:32:17.267102 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.266984    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kserve-provision-location\") pod \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") "
Sep 08 13:32:17.267102 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.267044    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80c21bdf-084d-4bd1-8c40-b3609b3334fc-cabundle-cert\") pod \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") "
Sep 08 13:32:17.267102 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.267083    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kube-rbac-proxy-sar-config\") pod \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\" (UID: \"80c21bdf-084d-4bd1-8c40-b3609b3334fc\") "
Sep 08 13:32:17.267356 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.267325    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "80c21bdf-084d-4bd1-8c40-b3609b3334fc" (UID: "80c21bdf-084d-4bd1-8c40-b3609b3334fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:32:17.267484 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.267460    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c21bdf-084d-4bd1-8c40-b3609b3334fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "80c21bdf-084d-4bd1-8c40-b3609b3334fc" (UID: "80c21bdf-084d-4bd1-8c40-b3609b3334fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:32:17.267549 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.267482    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "80c21bdf-084d-4bd1-8c40-b3609b3334fc" (UID: "80c21bdf-084d-4bd1-8c40-b3609b3334fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:32:17.269120 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.269095    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c21bdf-084d-4bd1-8c40-b3609b3334fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "80c21bdf-084d-4bd1-8c40-b3609b3334fc" (UID: "80c21bdf-084d-4bd1-8c40-b3609b3334fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:32:17.269248 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.269148    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kube-api-access-b2mjx" (OuterVolumeSpecName: "kube-api-access-b2mjx") pod "80c21bdf-084d-4bd1-8c40-b3609b3334fc" (UID: "80c21bdf-084d-4bd1-8c40-b3609b3334fc"). InnerVolumeSpecName "kube-api-access-b2mjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:32:17.368398 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.368310    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2mjx\" (UniqueName: \"kubernetes.io/projected/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kube-api-access-b2mjx\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:32:17.368398 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.368351    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80c21bdf-084d-4bd1-8c40-b3609b3334fc-proxy-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:32:17.368398 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.368365    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kserve-provision-location\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:32:17.368398 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.368378    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80c21bdf-084d-4bd1-8c40-b3609b3334fc-cabundle-cert\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:32:17.368398 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.368392    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80c21bdf-084d-4bd1-8c40-b3609b3334fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:32:17.882660 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.882603    2578 generic.go:356] "Generic (PLEG): container finished" podID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerID="6591fe2896b184523457cedd1b278a1432e8dbfd1a463b8b150becb3f70a5441" exitCode=0
Sep 08 13:32:17.883035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.882677    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" event={"ID":"80c21bdf-084d-4bd1-8c40-b3609b3334fc","Type":"ContainerDied","Data":"6591fe2896b184523457cedd1b278a1432e8dbfd1a463b8b150becb3f70a5441"}
Sep 08 13:32:17.883035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.882734    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr" event={"ID":"80c21bdf-084d-4bd1-8c40-b3609b3334fc","Type":"ContainerDied","Data":"e42e30c8049370ef43e8de2d44e428fbb050c050c21a2b660932a92d00aeb748"}
Sep 08 13:32:17.883035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.882681    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"
Sep 08 13:32:17.883035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.882761    2578 scope.go:117] "RemoveContainer" containerID="0e96481cc1401b2433294cb3c5971bd0a193ce2cdebec6b2e14a9a12fccaea37"
Sep 08 13:32:17.891317 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.891301    2578 scope.go:117] "RemoveContainer" containerID="6591fe2896b184523457cedd1b278a1432e8dbfd1a463b8b150becb3f70a5441"
Sep 08 13:32:17.898725 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.898703    2578 scope.go:117] "RemoveContainer" containerID="5dd7676b3dce6539f7a6cf0f0d0f064f3e7c66c2e0d2c03323db73758df227cb"
Sep 08 13:32:17.905513 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.905490    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"]
Sep 08 13:32:17.906035 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.906017    2578 scope.go:117] "RemoveContainer" containerID="0e96481cc1401b2433294cb3c5971bd0a193ce2cdebec6b2e14a9a12fccaea37"
Sep 08 13:32:17.906293 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:32:17.906270    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e96481cc1401b2433294cb3c5971bd0a193ce2cdebec6b2e14a9a12fccaea37\": container with ID starting with 0e96481cc1401b2433294cb3c5971bd0a193ce2cdebec6b2e14a9a12fccaea37 not found: ID does not exist" containerID="0e96481cc1401b2433294cb3c5971bd0a193ce2cdebec6b2e14a9a12fccaea37"
Sep 08 13:32:17.906339 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.906298    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e96481cc1401b2433294cb3c5971bd0a193ce2cdebec6b2e14a9a12fccaea37"} err="failed to get container status \"0e96481cc1401b2433294cb3c5971bd0a193ce2cdebec6b2e14a9a12fccaea37\": rpc error: code = NotFound desc = could not find container \"0e96481cc1401b2433294cb3c5971bd0a193ce2cdebec6b2e14a9a12fccaea37\": container with ID starting with 0e96481cc1401b2433294cb3c5971bd0a193ce2cdebec6b2e14a9a12fccaea37 not found: ID does not exist"
Sep 08 13:32:17.906339 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.906315    2578 scope.go:117] "RemoveContainer" containerID="6591fe2896b184523457cedd1b278a1432e8dbfd1a463b8b150becb3f70a5441"
Sep 08 13:32:17.906571 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:32:17.906554    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6591fe2896b184523457cedd1b278a1432e8dbfd1a463b8b150becb3f70a5441\": container with ID starting with 6591fe2896b184523457cedd1b278a1432e8dbfd1a463b8b150becb3f70a5441 not found: ID does not exist" containerID="6591fe2896b184523457cedd1b278a1432e8dbfd1a463b8b150becb3f70a5441"
Sep 08 13:32:17.906611 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.906581    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6591fe2896b184523457cedd1b278a1432e8dbfd1a463b8b150becb3f70a5441"} err="failed to get container status \"6591fe2896b184523457cedd1b278a1432e8dbfd1a463b8b150becb3f70a5441\": rpc error: code = NotFound desc = could not find container \"6591fe2896b184523457cedd1b278a1432e8dbfd1a463b8b150becb3f70a5441\": container with ID starting with 6591fe2896b184523457cedd1b278a1432e8dbfd1a463b8b150becb3f70a5441 not found: ID does not exist"
Sep 08 13:32:17.906611 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.906597    2578 scope.go:117] "RemoveContainer" containerID="5dd7676b3dce6539f7a6cf0f0d0f064f3e7c66c2e0d2c03323db73758df227cb"
Sep 08 13:32:17.906903 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:32:17.906884    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dd7676b3dce6539f7a6cf0f0d0f064f3e7c66c2e0d2c03323db73758df227cb\": container with ID starting with 5dd7676b3dce6539f7a6cf0f0d0f064f3e7c66c2e0d2c03323db73758df227cb not found: ID does not exist" containerID="5dd7676b3dce6539f7a6cf0f0d0f064f3e7c66c2e0d2c03323db73758df227cb"
Sep 08 13:32:17.906966 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.906910    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd7676b3dce6539f7a6cf0f0d0f064f3e7c66c2e0d2c03323db73758df227cb"} err="failed to get container status \"5dd7676b3dce6539f7a6cf0f0d0f064f3e7c66c2e0d2c03323db73758df227cb\": rpc error: code = NotFound desc = could not find container \"5dd7676b3dce6539f7a6cf0f0d0f064f3e7c66c2e0d2c03323db73758df227cb\": container with ID starting with 5dd7676b3dce6539f7a6cf0f0d0f064f3e7c66c2e0d2c03323db73758df227cb not found: ID does not exist"
Sep 08 13:32:17.909076 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:17.909054    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-3d858-predictor-55cb55b457-2sdfr"]
Sep 08 13:32:18.888865 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:18.888774    2578 generic.go:356] "Generic (PLEG): container finished" podID="34859624-406c-4dae-974c-89b725d34800" containerID="cd44002f5b23f866e8c6f5af3e8142e6244bf9a519bb96bcba36eba677446484" exitCode=0
Sep 08 13:32:18.888865 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:18.888842    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" event={"ID":"34859624-406c-4dae-974c-89b725d34800","Type":"ContainerDied","Data":"cd44002f5b23f866e8c6f5af3e8142e6244bf9a519bb96bcba36eba677446484"}
Sep 08 13:32:19.013657 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:19.013615    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" path="/var/lib/kubelet/pods/80c21bdf-084d-4bd1-8c40-b3609b3334fc/volumes"
Sep 08 13:32:19.896100 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:19.896062    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" event={"ID":"34859624-406c-4dae-974c-89b725d34800","Type":"ContainerStarted","Data":"b2673ba68b0ee836d3f7f3f0843ddeec4554cc508c40a8de006678e1047b836d"}
Sep 08 13:32:19.896596 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:19.896137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" event={"ID":"34859624-406c-4dae-974c-89b725d34800","Type":"ContainerStarted","Data":"e6856fd555bfdb56e2713938896657e370397c42656d88fbfcb47a7a8cc38fe9"}
Sep 08 13:32:19.896596 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:19.896355    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:19.916805 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:19.916763    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" podStartSLOduration=3.229826236 podStartE2EDuration="6.91675051s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.686924274s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:32:19.91595802 +0000 UTC m=+957.516055660" watchObservedRunningTime="2026-09-08 13:32:19.91675051 +0000 UTC m=+957.516848193"
Sep 08 13:32:20.901291 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:20.901246    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:20.902549 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:20.902518    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 08 13:32:21.905935 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:21.905890    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 08 13:32:25.808471 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:25.808434    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:32:26.813416 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:26.813366    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:32:26.910516 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:26.910488    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:32:26.911003 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:26.910978    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 08 13:32:35.808337 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:35.808297    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:32:36.812595 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:36.812553    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:32:36.911471 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:36.911434    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 08 13:32:45.809231 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:45.809142    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:32:46.812583 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:46.812545    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:32:46.911114 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:46.911077    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 08 13:32:55.808980 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:55.808933    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:32:56.813233 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:56.813194    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:32:56.911584 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:32:56.911548    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 08 13:33:05.808693 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:05.808653    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:33:06.812916 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:06.812875    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:33:06.910981 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:06.910949    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 08 13:33:15.808833 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:15.808799    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:33:16.813778 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:16.813746    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:33:16.911461 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:16.911421    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 08 13:33:23.154241 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:23.154205    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"]
Sep 08 13:33:23.154850 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:23.154497    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kserve-container" containerID="cri-o://bca0c27764f77908cd93f8157f5344aa9c08fb528f309022cff101b117ee9b4a" gracePeriod=30
Sep 08 13:33:23.154850 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:23.154548    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kube-rbac-proxy" containerID="cri-o://fbe703a741b4ed119f4f51486100a328e7e91160ed19a4c3c8ce647d912735f6" gracePeriod=30
Sep 08 13:33:23.187773 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:23.187747    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"]
Sep 08 13:33:23.188012 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:23.187987    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kserve-container" containerID="cri-o://dafdfb4f7cbeb64513bd25c7533a38ef4f16cf900794e4aa0c52b55751723196" gracePeriod=30
Sep 08 13:33:23.188118 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:23.188013    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kube-rbac-proxy" containerID="cri-o://481a9f90c183deb8f77855dfc5fccc0025ed4030a884a017162b81bf46de486c" gracePeriod=30
Sep 08 13:33:24.171258 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:24.171225    2578 generic.go:356] "Generic (PLEG): container finished" podID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerID="481a9f90c183deb8f77855dfc5fccc0025ed4030a884a017162b81bf46de486c" exitCode=2
Sep 08 13:33:24.171692 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:24.171289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" event={"ID":"22262be0-d3bc-4a39-b2a5-b0ae2d124092","Type":"ContainerDied","Data":"481a9f90c183deb8f77855dfc5fccc0025ed4030a884a017162b81bf46de486c"}
Sep 08 13:33:24.173721 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:24.173697    2578 generic.go:356] "Generic (PLEG): container finished" podID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerID="fbe703a741b4ed119f4f51486100a328e7e91160ed19a4c3c8ce647d912735f6" exitCode=2
Sep 08 13:33:24.173826 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:24.173738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" event={"ID":"eb649e86-0ada-48fb-8e65-d9ee1504b5d0","Type":"ContainerDied","Data":"fbe703a741b4ed119f4f51486100a328e7e91160ed19a4c3c8ce647d912735f6"}
Sep 08 13:33:25.803431 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:25.803391    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 08 13:33:25.808612 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:25.808584    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 08 13:33:26.808281 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:26.808237    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 08 13:33:26.812658 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:26.812613    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 08 13:33:26.911830 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:26.911800    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:33:27.877769 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:27.877747    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:33:27.988280 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:27.988252    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:33:28.061102 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.061068    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kserve-provision-location\") pod \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") "
Sep 08 13:33:28.061297 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.061122    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kube-rbac-proxy-sar-config\") pod \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") "
Sep 08 13:33:28.061297 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.061151    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22262be0-d3bc-4a39-b2a5-b0ae2d124092-proxy-tls\") pod \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") "
Sep 08 13:33:28.061297 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.061191    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbdc6\" (UniqueName: \"kubernetes.io/projected/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kube-api-access-rbdc6\") pod \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") "
Sep 08 13:33:28.061297 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.061223    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22262be0-d3bc-4a39-b2a5-b0ae2d124092-cabundle-cert\") pod \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\" (UID: \"22262be0-d3bc-4a39-b2a5-b0ae2d124092\") "
Sep 08 13:33:28.061532 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.061459    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22262be0-d3bc-4a39-b2a5-b0ae2d124092" (UID: "22262be0-d3bc-4a39-b2a5-b0ae2d124092"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:33:28.061610 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.061579    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "22262be0-d3bc-4a39-b2a5-b0ae2d124092" (UID: "22262be0-d3bc-4a39-b2a5-b0ae2d124092"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:33:28.061711 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.061656    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22262be0-d3bc-4a39-b2a5-b0ae2d124092-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22262be0-d3bc-4a39-b2a5-b0ae2d124092" (UID: "22262be0-d3bc-4a39-b2a5-b0ae2d124092"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:33:28.063438 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.063405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kube-api-access-rbdc6" (OuterVolumeSpecName: "kube-api-access-rbdc6") pod "22262be0-d3bc-4a39-b2a5-b0ae2d124092" (UID: "22262be0-d3bc-4a39-b2a5-b0ae2d124092"). InnerVolumeSpecName "kube-api-access-rbdc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:33:28.063438 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.063409    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22262be0-d3bc-4a39-b2a5-b0ae2d124092-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "22262be0-d3bc-4a39-b2a5-b0ae2d124092" (UID: "22262be0-d3bc-4a39-b2a5-b0ae2d124092"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:33:28.162005 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.161978    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kube-rbac-proxy-sar-config\") pod \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") "
Sep 08 13:33:28.162164 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.162023    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-proxy-tls\") pod \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") "
Sep 08 13:33:28.162164 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.162082    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kserve-provision-location\") pod \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") "
Sep 08 13:33:28.162164 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.162098    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l42qc\" (UniqueName: \"kubernetes.io/projected/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kube-api-access-l42qc\") pod \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") "
Sep 08 13:33:28.162164 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.162113    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-cabundle-cert\") pod \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\" (UID: \"eb649e86-0ada-48fb-8e65-d9ee1504b5d0\") "
Sep 08 13:33:28.162394 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.162313    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kserve-provision-location\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:33:28.162394 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.162331    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:33:28.162394 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.162345    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22262be0-d3bc-4a39-b2a5-b0ae2d124092-proxy-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:33:28.162394 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.162360    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbdc6\" (UniqueName: \"kubernetes.io/projected/22262be0-d3bc-4a39-b2a5-b0ae2d124092-kube-api-access-rbdc6\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:33:28.162394 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.162376    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22262be0-d3bc-4a39-b2a5-b0ae2d124092-cabundle-cert\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:33:28.162646 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.162400    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb649e86-0ada-48fb-8e65-d9ee1504b5d0" (UID: "eb649e86-0ada-48fb-8e65-d9ee1504b5d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:33:28.162646 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.162420    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb649e86-0ada-48fb-8e65-d9ee1504b5d0" (UID: "eb649e86-0ada-48fb-8e65-d9ee1504b5d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:33:28.162646 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.162517    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb649e86-0ada-48fb-8e65-d9ee1504b5d0" (UID: "eb649e86-0ada-48fb-8e65-d9ee1504b5d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:33:28.164077 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.164054    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kube-api-access-l42qc" (OuterVolumeSpecName: "kube-api-access-l42qc") pod "eb649e86-0ada-48fb-8e65-d9ee1504b5d0" (UID: "eb649e86-0ada-48fb-8e65-d9ee1504b5d0"). InnerVolumeSpecName "kube-api-access-l42qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:33:28.164187 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.164169    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb649e86-0ada-48fb-8e65-d9ee1504b5d0" (UID: "eb649e86-0ada-48fb-8e65-d9ee1504b5d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:33:28.197214 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.197187    2578 generic.go:356] "Generic (PLEG): container finished" podID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerID="bca0c27764f77908cd93f8157f5344aa9c08fb528f309022cff101b117ee9b4a" exitCode=0
Sep 08 13:33:28.197323 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.197247    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"
Sep 08 13:33:28.197323 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.197264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" event={"ID":"eb649e86-0ada-48fb-8e65-d9ee1504b5d0","Type":"ContainerDied","Data":"bca0c27764f77908cd93f8157f5344aa9c08fb528f309022cff101b117ee9b4a"}
Sep 08 13:33:28.197412 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.197331    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5" event={"ID":"eb649e86-0ada-48fb-8e65-d9ee1504b5d0","Type":"ContainerDied","Data":"d18a4cb93d72f08aab1dd30d6ae2587cd1ee66efc478c5133a7d971c1adcd459"}
Sep 08 13:33:28.197412 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.197371    2578 scope.go:117] "RemoveContainer" containerID="fbe703a741b4ed119f4f51486100a328e7e91160ed19a4c3c8ce647d912735f6"
Sep 08 13:33:28.199465 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.199446    2578 generic.go:356] "Generic (PLEG): container finished" podID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerID="dafdfb4f7cbeb64513bd25c7533a38ef4f16cf900794e4aa0c52b55751723196" exitCode=0
Sep 08 13:33:28.199566 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.199481    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" event={"ID":"22262be0-d3bc-4a39-b2a5-b0ae2d124092","Type":"ContainerDied","Data":"dafdfb4f7cbeb64513bd25c7533a38ef4f16cf900794e4aa0c52b55751723196"}
Sep 08 13:33:28.199566 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.199522    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2" event={"ID":"22262be0-d3bc-4a39-b2a5-b0ae2d124092","Type":"ContainerDied","Data":"03105da687db624d2248da0ada7199323dcea938fdf00f3b6bb77a74e4add611"}
Sep 08 13:33:28.199566 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.199525    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"
Sep 08 13:33:28.207241 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.207220    2578 scope.go:117] "RemoveContainer" containerID="bca0c27764f77908cd93f8157f5344aa9c08fb528f309022cff101b117ee9b4a"
Sep 08 13:33:28.214796 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.214778    2578 scope.go:117] "RemoveContainer" containerID="de179efe4e6482618cda131bbb87eb3fe9b03aa272913d453e06e3c58fd5b4cb"
Sep 08 13:33:28.220468 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.220449    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"]
Sep 08 13:33:28.222648 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.222527    2578 scope.go:117] "RemoveContainer" containerID="fbe703a741b4ed119f4f51486100a328e7e91160ed19a4c3c8ce647d912735f6"
Sep 08 13:33:28.223029 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:33:28.223004    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe703a741b4ed119f4f51486100a328e7e91160ed19a4c3c8ce647d912735f6\": container with ID starting with fbe703a741b4ed119f4f51486100a328e7e91160ed19a4c3c8ce647d912735f6 not found: ID does not exist" containerID="fbe703a741b4ed119f4f51486100a328e7e91160ed19a4c3c8ce647d912735f6"
Sep 08 13:33:28.223136 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.223050    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe703a741b4ed119f4f51486100a328e7e91160ed19a4c3c8ce647d912735f6"} err="failed to get container status \"fbe703a741b4ed119f4f51486100a328e7e91160ed19a4c3c8ce647d912735f6\": rpc error: code = NotFound desc = could not find container \"fbe703a741b4ed119f4f51486100a328e7e91160ed19a4c3c8ce647d912735f6\": container with ID starting with fbe703a741b4ed119f4f51486100a328e7e91160ed19a4c3c8ce647d912735f6 not found: ID does not exist"
Sep 08 13:33:28.223136 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.223067    2578 scope.go:117] "RemoveContainer" containerID="bca0c27764f77908cd93f8157f5344aa9c08fb528f309022cff101b117ee9b4a"
Sep 08 13:33:28.223440 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:33:28.223343    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca0c27764f77908cd93f8157f5344aa9c08fb528f309022cff101b117ee9b4a\": container with ID starting with bca0c27764f77908cd93f8157f5344aa9c08fb528f309022cff101b117ee9b4a not found: ID does not exist" containerID="bca0c27764f77908cd93f8157f5344aa9c08fb528f309022cff101b117ee9b4a"
Sep 08 13:33:28.223440 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.223371    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca0c27764f77908cd93f8157f5344aa9c08fb528f309022cff101b117ee9b4a"} err="failed to get container status \"bca0c27764f77908cd93f8157f5344aa9c08fb528f309022cff101b117ee9b4a\": rpc error: code = NotFound desc = could not find container \"bca0c27764f77908cd93f8157f5344aa9c08fb528f309022cff101b117ee9b4a\": container with ID starting with bca0c27764f77908cd93f8157f5344aa9c08fb528f309022cff101b117ee9b4a not found: ID does not exist"
Sep 08 13:33:28.223440 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.223389    2578 scope.go:117] "RemoveContainer" containerID="de179efe4e6482618cda131bbb87eb3fe9b03aa272913d453e06e3c58fd5b4cb"
Sep 08 13:33:28.223746 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:33:28.223721    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de179efe4e6482618cda131bbb87eb3fe9b03aa272913d453e06e3c58fd5b4cb\": container with ID starting with de179efe4e6482618cda131bbb87eb3fe9b03aa272913d453e06e3c58fd5b4cb not found: ID does not exist" containerID="de179efe4e6482618cda131bbb87eb3fe9b03aa272913d453e06e3c58fd5b4cb"
Sep 08 13:33:28.223810 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.223754    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de179efe4e6482618cda131bbb87eb3fe9b03aa272913d453e06e3c58fd5b4cb"} err="failed to get container status \"de179efe4e6482618cda131bbb87eb3fe9b03aa272913d453e06e3c58fd5b4cb\": rpc error: code = NotFound desc = could not find container \"de179efe4e6482618cda131bbb87eb3fe9b03aa272913d453e06e3c58fd5b4cb\": container with ID starting with de179efe4e6482618cda131bbb87eb3fe9b03aa272913d453e06e3c58fd5b4cb not found: ID does not exist"
Sep 08 13:33:28.223810 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.223776    2578 scope.go:117] "RemoveContainer" containerID="481a9f90c183deb8f77855dfc5fccc0025ed4030a884a017162b81bf46de486c"
Sep 08 13:33:28.225070 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.225050    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-l4xk5"]
Sep 08 13:33:28.230811 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.230796    2578 scope.go:117] "RemoveContainer" containerID="dafdfb4f7cbeb64513bd25c7533a38ef4f16cf900794e4aa0c52b55751723196"
Sep 08 13:33:28.236457 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.236434    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"]
Sep 08 13:33:28.238714 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.238661    2578 scope.go:117] "RemoveContainer" containerID="cfae7d2532149f42c7453a706487387fa5a1d0a409430b8c38b4db5f121f8b6a"
Sep 08 13:33:28.240001 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.239983    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-q8ms2"]
Sep 08 13:33:28.245819 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.245804    2578 scope.go:117] "RemoveContainer" containerID="481a9f90c183deb8f77855dfc5fccc0025ed4030a884a017162b81bf46de486c"
Sep 08 13:33:28.246084 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:33:28.246066    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"481a9f90c183deb8f77855dfc5fccc0025ed4030a884a017162b81bf46de486c\": container with ID starting with 481a9f90c183deb8f77855dfc5fccc0025ed4030a884a017162b81bf46de486c not found: ID does not exist" containerID="481a9f90c183deb8f77855dfc5fccc0025ed4030a884a017162b81bf46de486c"
Sep 08 13:33:28.246132 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.246089    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"481a9f90c183deb8f77855dfc5fccc0025ed4030a884a017162b81bf46de486c"} err="failed to get container status \"481a9f90c183deb8f77855dfc5fccc0025ed4030a884a017162b81bf46de486c\": rpc error: code = NotFound desc = could not find container \"481a9f90c183deb8f77855dfc5fccc0025ed4030a884a017162b81bf46de486c\": container with ID starting with 481a9f90c183deb8f77855dfc5fccc0025ed4030a884a017162b81bf46de486c not found: ID does not exist"
Sep 08 13:33:28.246132 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.246104    2578 scope.go:117] "RemoveContainer" containerID="dafdfb4f7cbeb64513bd25c7533a38ef4f16cf900794e4aa0c52b55751723196"
Sep 08 13:33:28.246328 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:33:28.246312    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dafdfb4f7cbeb64513bd25c7533a38ef4f16cf900794e4aa0c52b55751723196\": container with ID starting with dafdfb4f7cbeb64513bd25c7533a38ef4f16cf900794e4aa0c52b55751723196 not found: ID does not exist" containerID="dafdfb4f7cbeb64513bd25c7533a38ef4f16cf900794e4aa0c52b55751723196"
Sep 08 13:33:28.246392 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.246338    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dafdfb4f7cbeb64513bd25c7533a38ef4f16cf900794e4aa0c52b55751723196"} err="failed to get container status \"dafdfb4f7cbeb64513bd25c7533a38ef4f16cf900794e4aa0c52b55751723196\": rpc error: code = NotFound desc = could not find container \"dafdfb4f7cbeb64513bd25c7533a38ef4f16cf900794e4aa0c52b55751723196\": container with ID starting with dafdfb4f7cbeb64513bd25c7533a38ef4f16cf900794e4aa0c52b55751723196 not found: ID does not exist"
Sep 08 13:33:28.246392 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.246362    2578 scope.go:117] "RemoveContainer" containerID="cfae7d2532149f42c7453a706487387fa5a1d0a409430b8c38b4db5f121f8b6a"
Sep 08 13:33:28.246591 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:33:28.246575    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfae7d2532149f42c7453a706487387fa5a1d0a409430b8c38b4db5f121f8b6a\": container with ID starting with cfae7d2532149f42c7453a706487387fa5a1d0a409430b8c38b4db5f121f8b6a not found: ID does not exist" containerID="cfae7d2532149f42c7453a706487387fa5a1d0a409430b8c38b4db5f121f8b6a"
Sep 08 13:33:28.246669 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.246594    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfae7d2532149f42c7453a706487387fa5a1d0a409430b8c38b4db5f121f8b6a"} err="failed to get container status \"cfae7d2532149f42c7453a706487387fa5a1d0a409430b8c38b4db5f121f8b6a\": rpc error: code = NotFound desc = could not find container \"cfae7d2532149f42c7453a706487387fa5a1d0a409430b8c38b4db5f121f8b6a\": container with ID starting with cfae7d2532149f42c7453a706487387fa5a1d0a409430b8c38b4db5f121f8b6a not found: ID does not exist"
Sep 08 13:33:28.263029 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.263006    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:33:28.263029 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.263026    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-proxy-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:33:28.263148 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.263036    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kserve-provision-location\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:33:28.263148 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.263046    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l42qc\" (UniqueName: \"kubernetes.io/projected/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-kube-api-access-l42qc\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:33:28.263148 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:28.263055    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb649e86-0ada-48fb-8e65-d9ee1504b5d0-cabundle-cert\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:33:29.013324 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:29.013237    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" path="/var/lib/kubelet/pods/22262be0-d3bc-4a39-b2a5-b0ae2d124092/volumes"
Sep 08 13:33:29.013911 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:33:29.013891    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" path="/var/lib/kubelet/pods/eb649e86-0ada-48fb-8e65-d9ee1504b5d0/volumes"
Sep 08 13:34:03.833680 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.833642    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"]
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.833961    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kube-rbac-proxy"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.833973    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kube-rbac-proxy"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.833983    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kserve-container"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.833989    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kserve-container"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.833995    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kube-rbac-proxy"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834001    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kube-rbac-proxy"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834014    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kserve-container"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834020    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kserve-container"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834027    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kserve-container"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834032    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kserve-container"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834040    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="storage-initializer"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834045    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="storage-initializer"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834053    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="storage-initializer"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834059    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="storage-initializer"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834066    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="storage-initializer"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834071    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="storage-initializer"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834082    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kube-rbac-proxy"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834088    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kube-rbac-proxy"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834138    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kserve-container"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834144    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kube-rbac-proxy"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834152    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="80c21bdf-084d-4bd1-8c40-b3609b3334fc" containerName="kube-rbac-proxy"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834158    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kserve-container"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834164    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="22262be0-d3bc-4a39-b2a5-b0ae2d124092" containerName="kserve-container"
Sep 08 13:34:03.834376 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.834170    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb649e86-0ada-48fb-8e65-d9ee1504b5d0" containerName="kube-rbac-proxy"
Sep 08 13:34:03.837166 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.837149    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:03.840959 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.840938    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-27b1d-predictor-serving-cert\""
Sep 08 13:34:03.841230 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.841210    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-27b1d-kube-rbac-proxy-sar-config\""
Sep 08 13:34:03.856416 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.856391    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"]
Sep 08 13:34:03.888046 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.888015    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"]
Sep 08 13:34:03.888310 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.888284    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kube-rbac-proxy" containerID="cri-o://b2673ba68b0ee836d3f7f3f0843ddeec4554cc508c40a8de006678e1047b836d" gracePeriod=30
Sep 08 13:34:03.888310 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.888279    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kserve-container" containerID="cri-o://e6856fd555bfdb56e2713938896657e370397c42656d88fbfcb47a7a8cc38fe9" gracePeriod=30
Sep 08 13:34:03.957041 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.957007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb474c84-e871-489e-8bc9-6e8cba8adcb5-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr\" (UID: \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\") " pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:03.957191 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.957071    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb474c84-e871-489e-8bc9-6e8cba8adcb5-proxy-tls\") pod \"message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr\" (UID: \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\") " pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:03.957191 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:03.957142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txsdb\" (UniqueName: \"kubernetes.io/projected/cb474c84-e871-489e-8bc9-6e8cba8adcb5-kube-api-access-txsdb\") pod \"message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr\" (UID: \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\") " pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:04.058099 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:04.058065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb474c84-e871-489e-8bc9-6e8cba8adcb5-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr\" (UID: \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\") " pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:04.058270 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:04.058222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb474c84-e871-489e-8bc9-6e8cba8adcb5-proxy-tls\") pod \"message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr\" (UID: \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\") " pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:04.058330 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:04.058296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txsdb\" (UniqueName: \"kubernetes.io/projected/cb474c84-e871-489e-8bc9-6e8cba8adcb5-kube-api-access-txsdb\") pod \"message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr\" (UID: \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\") " pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:04.058403 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:34:04.058387    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-serving-cert: secret "message-dumper-raw-27b1d-predictor-serving-cert" not found
Sep 08 13:34:04.058456 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:34:04.058446    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb474c84-e871-489e-8bc9-6e8cba8adcb5-proxy-tls podName:cb474c84-e871-489e-8bc9-6e8cba8adcb5 nodeName:}" failed. No retries permitted until 2026-09-08 13:34:04.558430966 +0000 UTC m=+1062.158528576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cb474c84-e871-489e-8bc9-6e8cba8adcb5-proxy-tls") pod "message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr" (UID: "cb474c84-e871-489e-8bc9-6e8cba8adcb5") : secret "message-dumper-raw-27b1d-predictor-serving-cert" not found
Sep 08 13:34:04.058855 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:04.058836    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb474c84-e871-489e-8bc9-6e8cba8adcb5-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr\" (UID: \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\") " pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:04.071564 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:04.071543    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txsdb\" (UniqueName: \"kubernetes.io/projected/cb474c84-e871-489e-8bc9-6e8cba8adcb5-kube-api-access-txsdb\") pod \"message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr\" (UID: \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\") " pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:04.352527 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:04.352493    2578 generic.go:356] "Generic (PLEG): container finished" podID="34859624-406c-4dae-974c-89b725d34800" containerID="b2673ba68b0ee836d3f7f3f0843ddeec4554cc508c40a8de006678e1047b836d" exitCode=2
Sep 08 13:34:04.352722 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:04.352545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" event={"ID":"34859624-406c-4dae-974c-89b725d34800","Type":"ContainerDied","Data":"b2673ba68b0ee836d3f7f3f0843ddeec4554cc508c40a8de006678e1047b836d"}
Sep 08 13:34:04.562446 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:04.562408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb474c84-e871-489e-8bc9-6e8cba8adcb5-proxy-tls\") pod \"message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr\" (UID: \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\") " pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:04.564802 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:04.564779    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb474c84-e871-489e-8bc9-6e8cba8adcb5-proxy-tls\") pod \"message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr\" (UID: \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\") " pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:04.749588 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:04.749543    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:05.020232 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:05.020204    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"]
Sep 08 13:34:05.022106 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:34:05.022081    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb474c84_e871_489e_8bc9_6e8cba8adcb5.slice/crio-c3fee98a481aa7dc82a5d36ba8aaa8cb8584ee0cf7a81439398c3d3bdf24a44b WatchSource:0}: Error finding container c3fee98a481aa7dc82a5d36ba8aaa8cb8584ee0cf7a81439398c3d3bdf24a44b: Status 404 returned error can't find the container with id c3fee98a481aa7dc82a5d36ba8aaa8cb8584ee0cf7a81439398c3d3bdf24a44b
Sep 08 13:34:05.023965 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:05.023949    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:34:05.357833 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:05.357745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr" event={"ID":"cb474c84-e871-489e-8bc9-6e8cba8adcb5","Type":"ContainerStarted","Data":"c3fee98a481aa7dc82a5d36ba8aaa8cb8584ee0cf7a81439398c3d3bdf24a44b"}
Sep 08 13:34:06.906400 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:06.906358    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 08 13:34:06.911747 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:06.911720    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 08 13:34:07.370106 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.370007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr" event={"ID":"cb474c84-e871-489e-8bc9-6e8cba8adcb5","Type":"ContainerStarted","Data":"aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96"}
Sep 08 13:34:07.370106 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.370074    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr" event={"ID":"cb474c84-e871-489e-8bc9-6e8cba8adcb5","Type":"ContainerStarted","Data":"0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf"}
Sep 08 13:34:07.370106 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.370107    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:07.370349 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.370135    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:07.371789 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.371763    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:07.391432 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.391385    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr" podStartSLOduration=3.036958757 podStartE2EDuration="4.39136843s" totalImagesPullingTime="1.354409673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:34:07.390677003 +0000 UTC m=+1064.990774643" watchObservedRunningTime="2026-09-08 13:34:07.39136843 +0000 UTC m=+1064.991466057"
Sep 08 13:34:07.528263 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.528238    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:34:07.688374 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.688331    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34859624-406c-4dae-974c-89b725d34800-kserve-provision-location\") pod \"34859624-406c-4dae-974c-89b725d34800\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") "
Sep 08 13:34:07.688573 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.688403    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34859624-406c-4dae-974c-89b725d34800-proxy-tls\") pod \"34859624-406c-4dae-974c-89b725d34800\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") "
Sep 08 13:34:07.688573 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.688486    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34859624-406c-4dae-974c-89b725d34800-cabundle-cert\") pod \"34859624-406c-4dae-974c-89b725d34800\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") "
Sep 08 13:34:07.688573 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.688528    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4z9d\" (UniqueName: \"kubernetes.io/projected/34859624-406c-4dae-974c-89b725d34800-kube-api-access-d4z9d\") pod \"34859624-406c-4dae-974c-89b725d34800\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") "
Sep 08 13:34:07.688573 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.688555    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34859624-406c-4dae-974c-89b725d34800-kube-rbac-proxy-sar-config\") pod \"34859624-406c-4dae-974c-89b725d34800\" (UID: \"34859624-406c-4dae-974c-89b725d34800\") "
Sep 08 13:34:07.688840 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.688652    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34859624-406c-4dae-974c-89b725d34800-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34859624-406c-4dae-974c-89b725d34800" (UID: "34859624-406c-4dae-974c-89b725d34800"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:07.688911 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.688894    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34859624-406c-4dae-974c-89b725d34800-kserve-provision-location\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:34:07.688969 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.688928    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34859624-406c-4dae-974c-89b725d34800-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34859624-406c-4dae-974c-89b725d34800" (UID: "34859624-406c-4dae-974c-89b725d34800"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:34:07.688969 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.688952    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34859624-406c-4dae-974c-89b725d34800-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34859624-406c-4dae-974c-89b725d34800" (UID: "34859624-406c-4dae-974c-89b725d34800"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:34:07.690537 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.690514    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34859624-406c-4dae-974c-89b725d34800-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34859624-406c-4dae-974c-89b725d34800" (UID: "34859624-406c-4dae-974c-89b725d34800"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:34:07.690636 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.690535    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34859624-406c-4dae-974c-89b725d34800-kube-api-access-d4z9d" (OuterVolumeSpecName: "kube-api-access-d4z9d") pod "34859624-406c-4dae-974c-89b725d34800" (UID: "34859624-406c-4dae-974c-89b725d34800"). InnerVolumeSpecName "kube-api-access-d4z9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:34:07.789895 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.789866    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34859624-406c-4dae-974c-89b725d34800-cabundle-cert\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:34:07.789895 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.789891    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4z9d\" (UniqueName: \"kubernetes.io/projected/34859624-406c-4dae-974c-89b725d34800-kube-api-access-d4z9d\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:34:07.789895 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.789900    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34859624-406c-4dae-974c-89b725d34800-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:34:07.790098 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:07.789911    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34859624-406c-4dae-974c-89b725d34800-proxy-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:34:08.376839 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:08.376807    2578 generic.go:356] "Generic (PLEG): container finished" podID="34859624-406c-4dae-974c-89b725d34800" containerID="e6856fd555bfdb56e2713938896657e370397c42656d88fbfcb47a7a8cc38fe9" exitCode=0
Sep 08 13:34:08.377301 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:08.376865    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"
Sep 08 13:34:08.377301 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:08.376880    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" event={"ID":"34859624-406c-4dae-974c-89b725d34800","Type":"ContainerDied","Data":"e6856fd555bfdb56e2713938896657e370397c42656d88fbfcb47a7a8cc38fe9"}
Sep 08 13:34:08.377301 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:08.376949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq" event={"ID":"34859624-406c-4dae-974c-89b725d34800","Type":"ContainerDied","Data":"7cc6e48b163ba8980cf51bd8e92c444d24ae368e8e32ff5a095cacf16e7382ce"}
Sep 08 13:34:08.377301 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:08.376977    2578 scope.go:117] "RemoveContainer" containerID="b2673ba68b0ee836d3f7f3f0843ddeec4554cc508c40a8de006678e1047b836d"
Sep 08 13:34:08.385563 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:08.385545    2578 scope.go:117] "RemoveContainer" containerID="e6856fd555bfdb56e2713938896657e370397c42656d88fbfcb47a7a8cc38fe9"
Sep 08 13:34:08.393009 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:08.392993    2578 scope.go:117] "RemoveContainer" containerID="cd44002f5b23f866e8c6f5af3e8142e6244bf9a519bb96bcba36eba677446484"
Sep 08 13:34:08.400860 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:08.400540    2578 scope.go:117] "RemoveContainer" containerID="b2673ba68b0ee836d3f7f3f0843ddeec4554cc508c40a8de006678e1047b836d"
Sep 08 13:34:08.400860 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:34:08.400835    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2673ba68b0ee836d3f7f3f0843ddeec4554cc508c40a8de006678e1047b836d\": container with ID starting with b2673ba68b0ee836d3f7f3f0843ddeec4554cc508c40a8de006678e1047b836d not found: ID does not exist" containerID="b2673ba68b0ee836d3f7f3f0843ddeec4554cc508c40a8de006678e1047b836d"
Sep 08 13:34:08.400983 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:08.400860    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2673ba68b0ee836d3f7f3f0843ddeec4554cc508c40a8de006678e1047b836d"} err="failed to get container status \"b2673ba68b0ee836d3f7f3f0843ddeec4554cc508c40a8de006678e1047b836d\": rpc error: code = NotFound desc = could not find container \"b2673ba68b0ee836d3f7f3f0843ddeec4554cc508c40a8de006678e1047b836d\": container with ID starting with b2673ba68b0ee836d3f7f3f0843ddeec4554cc508c40a8de006678e1047b836d not found: ID does not exist"
Sep 08 13:34:08.400983 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:08.400886    2578 scope.go:117] "RemoveContainer" containerID="e6856fd555bfdb56e2713938896657e370397c42656d88fbfcb47a7a8cc38fe9"
Sep 08 13:34:08.400983 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:08.400923    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"]
Sep 08 13:34:08.401136 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:34:08.401119    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6856fd555bfdb56e2713938896657e370397c42656d88fbfcb47a7a8cc38fe9\": container with ID starting with e6856fd555bfdb56e2713938896657e370397c42656d88fbfcb47a7a8cc38fe9 not found: ID does not exist" containerID="e6856fd555bfdb56e2713938896657e370397c42656d88fbfcb47a7a8cc38fe9"
Sep 08 13:34:08.401176 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:08.401140    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6856fd555bfdb56e2713938896657e370397c42656d88fbfcb47a7a8cc38fe9"} err="failed to get container status \"e6856fd555bfdb56e2713938896657e370397c42656d88fbfcb47a7a8cc38fe9\": rpc error: code = NotFound desc = could not find container \"e6856fd555bfdb56e2713938896657e370397c42656d88fbfcb47a7a8cc38fe9\": container with ID starting with e6856fd555bfdb56e2713938896657e370397c42656d88fbfcb47a7a8cc38fe9 not found: ID does not exist"
Sep 08 13:34:08.401176 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:08.401154    2578 scope.go:117] "RemoveContainer" containerID="cd44002f5b23f866e8c6f5af3e8142e6244bf9a519bb96bcba36eba677446484"
Sep 08 13:34:08.401400 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:34:08.401383    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd44002f5b23f866e8c6f5af3e8142e6244bf9a519bb96bcba36eba677446484\": container with ID starting with cd44002f5b23f866e8c6f5af3e8142e6244bf9a519bb96bcba36eba677446484 not found: ID does not exist" containerID="cd44002f5b23f866e8c6f5af3e8142e6244bf9a519bb96bcba36eba677446484"
Sep 08 13:34:08.401455 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:08.401409    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd44002f5b23f866e8c6f5af3e8142e6244bf9a519bb96bcba36eba677446484"} err="failed to get container status \"cd44002f5b23f866e8c6f5af3e8142e6244bf9a519bb96bcba36eba677446484\": rpc error: code = NotFound desc = could not find container \"cd44002f5b23f866e8c6f5af3e8142e6244bf9a519bb96bcba36eba677446484\": container with ID starting with cd44002f5b23f866e8c6f5af3e8142e6244bf9a519bb96bcba36eba677446484 not found: ID does not exist"
Sep 08 13:34:08.406481 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:08.406459    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-90436-predictor-55d8dd6bc7-72zlq"]
Sep 08 13:34:09.012721 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:09.012688    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34859624-406c-4dae-974c-89b725d34800" path="/var/lib/kubelet/pods/34859624-406c-4dae-974c-89b725d34800/volumes"
Sep 08 13:34:14.385639 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:14.385597    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:34:23.788489 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.788457    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"]
Sep 08 13:34:23.788916 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.788797    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kserve-container"
Sep 08 13:34:23.788916 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.788810    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kserve-container"
Sep 08 13:34:23.788916 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.788828    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="storage-initializer"
Sep 08 13:34:23.788916 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.788834    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="storage-initializer"
Sep 08 13:34:23.788916 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.788847    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kube-rbac-proxy"
Sep 08 13:34:23.788916 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.788852    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kube-rbac-proxy"
Sep 08 13:34:23.788916 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.788911    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kserve-container"
Sep 08 13:34:23.788916 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.788919    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="34859624-406c-4dae-974c-89b725d34800" containerName="kube-rbac-proxy"
Sep 08 13:34:23.793588 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.793571    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:23.796061 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.796026    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:34:23.796238 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.796192    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:34:23.796336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.796244    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-27b1d-predictor-serving-cert\""
Sep 08 13:34:23.796458 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.796445    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-27b1d-kube-rbac-proxy-sar-config\""
Sep 08 13:34:23.802372 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.802351    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"]
Sep 08 13:34:23.923998 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.923967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37cde063-d68a-48ca-88db-7cf8a26b7b39-cabundle-cert\") pod \"isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") " pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:23.924188 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.924005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37cde063-d68a-48ca-88db-7cf8a26b7b39-kserve-provision-location\") pod \"isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") " pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:23.924188 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.924036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37cde063-d68a-48ca-88db-7cf8a26b7b39-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") " pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:23.924188 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.924055    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9llpj\" (UniqueName: \"kubernetes.io/projected/37cde063-d68a-48ca-88db-7cf8a26b7b39-kube-api-access-9llpj\") pod \"isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") " pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:23.924188 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:23.924173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37cde063-d68a-48ca-88db-7cf8a26b7b39-proxy-tls\") pod \"isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") " pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:24.025082 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:24.025047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37cde063-d68a-48ca-88db-7cf8a26b7b39-proxy-tls\") pod \"isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") " pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:24.025234 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:24.025195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37cde063-d68a-48ca-88db-7cf8a26b7b39-cabundle-cert\") pod \"isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") " pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:24.025234 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:24.025229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37cde063-d68a-48ca-88db-7cf8a26b7b39-kserve-provision-location\") pod \"isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") " pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:24.025403 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:24.025377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37cde063-d68a-48ca-88db-7cf8a26b7b39-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") " pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:24.025463 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:24.025428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9llpj\" (UniqueName: \"kubernetes.io/projected/37cde063-d68a-48ca-88db-7cf8a26b7b39-kube-api-access-9llpj\") pod \"isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") " pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:24.025664 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:24.025638    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37cde063-d68a-48ca-88db-7cf8a26b7b39-kserve-provision-location\") pod \"isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") " pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:24.026057 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:24.026033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37cde063-d68a-48ca-88db-7cf8a26b7b39-cabundle-cert\") pod \"isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") " pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:24.026152 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:24.026069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37cde063-d68a-48ca-88db-7cf8a26b7b39-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") " pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:24.027611 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:24.027584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37cde063-d68a-48ca-88db-7cf8a26b7b39-proxy-tls\") pod \"isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") " pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:24.035064 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:24.035040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9llpj\" (UniqueName: \"kubernetes.io/projected/37cde063-d68a-48ca-88db-7cf8a26b7b39-kube-api-access-9llpj\") pod \"isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") " pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:24.104759 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:24.104678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:24.383761 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:24.383736    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"]
Sep 08 13:34:24.386831 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:34:24.386793    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cde063_d68a_48ca_88db_7cf8a26b7b39.slice/crio-201833d27aec92d2842f7e80e9fde522b6f5600ab2a7dc9c4b8236c607527b75 WatchSource:0}: Error finding container 201833d27aec92d2842f7e80e9fde522b6f5600ab2a7dc9c4b8236c607527b75: Status 404 returned error can't find the container with id 201833d27aec92d2842f7e80e9fde522b6f5600ab2a7dc9c4b8236c607527b75
Sep 08 13:34:24.442776 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:24.442746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" event={"ID":"37cde063-d68a-48ca-88db-7cf8a26b7b39","Type":"ContainerStarted","Data":"201833d27aec92d2842f7e80e9fde522b6f5600ab2a7dc9c4b8236c607527b75"}
Sep 08 13:34:25.447777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:25.447737    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" event={"ID":"37cde063-d68a-48ca-88db-7cf8a26b7b39","Type":"ContainerStarted","Data":"50210e5e179099070e1602a3f9b45e8792b45b6a057a62a0cfaeb44ce882b040"}
Sep 08 13:34:28.462005 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:28.461911    2578 generic.go:356] "Generic (PLEG): container finished" podID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerID="50210e5e179099070e1602a3f9b45e8792b45b6a057a62a0cfaeb44ce882b040" exitCode=0
Sep 08 13:34:28.462005 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:28.461987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" event={"ID":"37cde063-d68a-48ca-88db-7cf8a26b7b39","Type":"ContainerDied","Data":"50210e5e179099070e1602a3f9b45e8792b45b6a057a62a0cfaeb44ce882b040"}
Sep 08 13:34:29.468766 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:29.468732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" event={"ID":"37cde063-d68a-48ca-88db-7cf8a26b7b39","Type":"ContainerStarted","Data":"83147f9d7dddad86388b824a2f46d749dadb7fe77a3f5893d4421dddf042a08c"}
Sep 08 13:34:29.469130 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:29.468798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" event={"ID":"37cde063-d68a-48ca-88db-7cf8a26b7b39","Type":"ContainerStarted","Data":"eaeda957de2a41f78387f295c28f23afaf9c0ec0bc5b6af5c3001b81c2f46d37"}
Sep 08 13:34:29.469130 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:29.468822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" event={"ID":"37cde063-d68a-48ca-88db-7cf8a26b7b39","Type":"ContainerStarted","Data":"a919f789da0be64172477c87f1918fc26e84827afe1677e48223f02df316622e"}
Sep 08 13:34:29.469130 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:29.469105    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:29.469297 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:29.469158    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:29.470597 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:29.470568    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:34:29.496082 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:29.496039    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podStartSLOduration=2.740050662 podStartE2EDuration="6.496026641s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.755975979s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:34:29.494536436 +0000 UTC m=+1087.094634062" watchObservedRunningTime="2026-09-08 13:34:29.496026641 +0000 UTC m=+1087.096124267"
Sep 08 13:34:30.473928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:30.473899    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:30.473928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:30.473899    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:34:30.475168 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:30.475137    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:34:31.478209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:31.478170    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:34:31.478820 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:31.478792    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:34:36.482530 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:36.482500    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:34:36.483132 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:36.482997    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:34:36.483404 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:36.483379    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:34:46.483758 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:46.483714    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:34:46.484192 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:46.484175    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:34:56.483253 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:56.483202    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:34:56.483684 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:34:56.483660    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:35:06.483391 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:06.483350    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:35:06.483923 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:06.483901    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:35:16.483874 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:16.483830    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:35:16.484433 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:16.484406    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:35:26.483839 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:26.483793    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:35:26.484343 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:26.484202    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:35:36.482979 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:36.482935    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:35:36.483480 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:36.483371    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:35:46.483696 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:46.483596    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:35:46.484186 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:46.483838    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:35:58.783586 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:58.783554    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr_cb474c84-e871-489e-8bc9-6e8cba8adcb5/kserve-container/0.log"
Sep 08 13:35:58.937738 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:58.937703    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"]
Sep 08 13:35:58.938315 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:58.938039    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container" containerID="cri-o://a919f789da0be64172477c87f1918fc26e84827afe1677e48223f02df316622e" gracePeriod=30
Sep 08 13:35:58.938315 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:58.938121    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kube-rbac-proxy" containerID="cri-o://eaeda957de2a41f78387f295c28f23afaf9c0ec0bc5b6af5c3001b81c2f46d37" gracePeriod=30
Sep 08 13:35:58.938955 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:58.938788    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent" containerID="cri-o://83147f9d7dddad86388b824a2f46d749dadb7fe77a3f5893d4421dddf042a08c" gracePeriod=30
Sep 08 13:35:59.072777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.072745    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"]
Sep 08 13:35:59.073069 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.073014    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr" podUID="cb474c84-e871-489e-8bc9-6e8cba8adcb5" containerName="kserve-container" containerID="cri-o://0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf" gracePeriod=30
Sep 08 13:35:59.073217 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.073019    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr" podUID="cb474c84-e871-489e-8bc9-6e8cba8adcb5" containerName="kube-rbac-proxy" containerID="cri-o://aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96" gracePeriod=30
Sep 08 13:35:59.323499 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.323475    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:35:59.342684 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.342662    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb474c84-e871-489e-8bc9-6e8cba8adcb5-proxy-tls\") pod \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\" (UID: \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\") "
Sep 08 13:35:59.342908 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.342760    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txsdb\" (UniqueName: \"kubernetes.io/projected/cb474c84-e871-489e-8bc9-6e8cba8adcb5-kube-api-access-txsdb\") pod \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\" (UID: \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\") "
Sep 08 13:35:59.343654 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.343373    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb474c84-e871-489e-8bc9-6e8cba8adcb5-kube-rbac-proxy-sar-config\") pod \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\" (UID: \"cb474c84-e871-489e-8bc9-6e8cba8adcb5\") "
Sep 08 13:35:59.343941 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.343916    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb474c84-e871-489e-8bc9-6e8cba8adcb5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb474c84-e871-489e-8bc9-6e8cba8adcb5" (UID: "cb474c84-e871-489e-8bc9-6e8cba8adcb5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:35:59.345082 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.345056    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb474c84-e871-489e-8bc9-6e8cba8adcb5-kube-api-access-txsdb" (OuterVolumeSpecName: "kube-api-access-txsdb") pod "cb474c84-e871-489e-8bc9-6e8cba8adcb5" (UID: "cb474c84-e871-489e-8bc9-6e8cba8adcb5"). InnerVolumeSpecName "kube-api-access-txsdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:35:59.345202 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.345183    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb474c84-e871-489e-8bc9-6e8cba8adcb5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb474c84-e871-489e-8bc9-6e8cba8adcb5" (UID: "cb474c84-e871-489e-8bc9-6e8cba8adcb5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:35:59.444857 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.444827    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb474c84-e871-489e-8bc9-6e8cba8adcb5-proxy-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:35:59.444857 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.444854    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txsdb\" (UniqueName: \"kubernetes.io/projected/cb474c84-e871-489e-8bc9-6e8cba8adcb5-kube-api-access-txsdb\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:35:59.444857 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.444864    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb474c84-e871-489e-8bc9-6e8cba8adcb5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:35:59.844782 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.844750    2578 generic.go:356] "Generic (PLEG): container finished" podID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerID="eaeda957de2a41f78387f295c28f23afaf9c0ec0bc5b6af5c3001b81c2f46d37" exitCode=2
Sep 08 13:35:59.845222 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.844823    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" event={"ID":"37cde063-d68a-48ca-88db-7cf8a26b7b39","Type":"ContainerDied","Data":"eaeda957de2a41f78387f295c28f23afaf9c0ec0bc5b6af5c3001b81c2f46d37"}
Sep 08 13:35:59.846430 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.846406    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb474c84-e871-489e-8bc9-6e8cba8adcb5" containerID="aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96" exitCode=2
Sep 08 13:35:59.846430 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.846430    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb474c84-e871-489e-8bc9-6e8cba8adcb5" containerID="0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf" exitCode=2
Sep 08 13:35:59.846610 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.846471    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"
Sep 08 13:35:59.846610 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.846477    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr" event={"ID":"cb474c84-e871-489e-8bc9-6e8cba8adcb5","Type":"ContainerDied","Data":"aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96"}
Sep 08 13:35:59.846610 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.846543    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr" event={"ID":"cb474c84-e871-489e-8bc9-6e8cba8adcb5","Type":"ContainerDied","Data":"0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf"}
Sep 08 13:35:59.846610 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.846569    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr" event={"ID":"cb474c84-e871-489e-8bc9-6e8cba8adcb5","Type":"ContainerDied","Data":"c3fee98a481aa7dc82a5d36ba8aaa8cb8584ee0cf7a81439398c3d3bdf24a44b"}
Sep 08 13:35:59.846610 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.846592    2578 scope.go:117] "RemoveContainer" containerID="aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96"
Sep 08 13:35:59.855822 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.855802    2578 scope.go:117] "RemoveContainer" containerID="0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf"
Sep 08 13:35:59.863202 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.863182    2578 scope.go:117] "RemoveContainer" containerID="aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96"
Sep 08 13:35:59.863470 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:35:59.863453    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96\": container with ID starting with aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96 not found: ID does not exist" containerID="aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96"
Sep 08 13:35:59.863511 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.863481    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96"} err="failed to get container status \"aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96\": rpc error: code = NotFound desc = could not find container \"aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96\": container with ID starting with aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96 not found: ID does not exist"
Sep 08 13:35:59.863511 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.863499    2578 scope.go:117] "RemoveContainer" containerID="0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf"
Sep 08 13:35:59.863787 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:35:59.863767    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf\": container with ID starting with 0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf not found: ID does not exist" containerID="0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf"
Sep 08 13:35:59.863822 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.863793    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf"} err="failed to get container status \"0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf\": rpc error: code = NotFound desc = could not find container \"0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf\": container with ID starting with 0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf not found: ID does not exist"
Sep 08 13:35:59.863822 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.863809    2578 scope.go:117] "RemoveContainer" containerID="aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96"
Sep 08 13:35:59.864045 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.864027    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96"} err="failed to get container status \"aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96\": rpc error: code = NotFound desc = could not find container \"aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96\": container with ID starting with aa6e93a7797bf4247bc9bfb0694c08e75849896fd3a13e16009dfd8fc8425e96 not found: ID does not exist"
Sep 08 13:35:59.864089 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.864047    2578 scope.go:117] "RemoveContainer" containerID="0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf"
Sep 08 13:35:59.864258 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.864243    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf"} err="failed to get container status \"0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf\": rpc error: code = NotFound desc = could not find container \"0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf\": container with ID starting with 0f3ed5315609a65ef5b983b119cc76b8e137f8f9ee4a83ea42479a959c91f1bf not found: ID does not exist"
Sep 08 13:35:59.868358 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.868339    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"]
Sep 08 13:35:59.874503 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:35:59.874480    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-27b1d-predictor-77fb46fc7d-wtfrr"]
Sep 08 13:36:01.012906 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:01.012866    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb474c84-e871-489e-8bc9-6e8cba8adcb5" path="/var/lib/kubelet/pods/cb474c84-e871-489e-8bc9-6e8cba8adcb5/volumes"
Sep 08 13:36:01.479249 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:01.479201    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 08 13:36:03.865242 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:03.865209    2578 generic.go:356] "Generic (PLEG): container finished" podID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerID="a919f789da0be64172477c87f1918fc26e84827afe1677e48223f02df316622e" exitCode=0
Sep 08 13:36:03.865611 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:03.865282    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" event={"ID":"37cde063-d68a-48ca-88db-7cf8a26b7b39","Type":"ContainerDied","Data":"a919f789da0be64172477c87f1918fc26e84827afe1677e48223f02df316622e"}
Sep 08 13:36:06.479097 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:06.479054    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 08 13:36:06.483468 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:06.483437    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:36:06.483782 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:06.483756    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:36:11.478869 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:11.478827    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 08 13:36:11.479268 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:11.478960    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:36:16.478544 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:16.478503    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 08 13:36:16.483906 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:16.483868    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:36:16.484233 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:16.484205    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:36:21.478840 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:21.478797    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 08 13:36:26.479317 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:26.479272    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 08 13:36:26.483626 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:26.483597    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 08 13:36:26.483784 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:26.483768    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:36:26.484064 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:26.484031    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:36:26.484151 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:26.484139    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:36:29.112572 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.112550    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:36:29.184024 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.183995    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37cde063-d68a-48ca-88db-7cf8a26b7b39-kserve-provision-location\") pod \"37cde063-d68a-48ca-88db-7cf8a26b7b39\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") "
Sep 08 13:36:29.184176 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.184050    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9llpj\" (UniqueName: \"kubernetes.io/projected/37cde063-d68a-48ca-88db-7cf8a26b7b39-kube-api-access-9llpj\") pod \"37cde063-d68a-48ca-88db-7cf8a26b7b39\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") "
Sep 08 13:36:29.184176 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.184099    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37cde063-d68a-48ca-88db-7cf8a26b7b39-cabundle-cert\") pod \"37cde063-d68a-48ca-88db-7cf8a26b7b39\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") "
Sep 08 13:36:29.184176 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.184123    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37cde063-d68a-48ca-88db-7cf8a26b7b39-kube-rbac-proxy-sar-config\") pod \"37cde063-d68a-48ca-88db-7cf8a26b7b39\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") "
Sep 08 13:36:29.184176 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.184147    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37cde063-d68a-48ca-88db-7cf8a26b7b39-proxy-tls\") pod \"37cde063-d68a-48ca-88db-7cf8a26b7b39\" (UID: \"37cde063-d68a-48ca-88db-7cf8a26b7b39\") "
Sep 08 13:36:29.184403 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.184310    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cde063-d68a-48ca-88db-7cf8a26b7b39-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37cde063-d68a-48ca-88db-7cf8a26b7b39" (UID: "37cde063-d68a-48ca-88db-7cf8a26b7b39"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:36:29.184466 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.184448    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cde063-d68a-48ca-88db-7cf8a26b7b39-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37cde063-d68a-48ca-88db-7cf8a26b7b39" (UID: "37cde063-d68a-48ca-88db-7cf8a26b7b39"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:36:29.184514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.184474    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cde063-d68a-48ca-88db-7cf8a26b7b39-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37cde063-d68a-48ca-88db-7cf8a26b7b39" (UID: "37cde063-d68a-48ca-88db-7cf8a26b7b39"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:36:29.184551 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.184517    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37cde063-d68a-48ca-88db-7cf8a26b7b39-kserve-provision-location\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:36:29.186179 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.186156    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37cde063-d68a-48ca-88db-7cf8a26b7b39-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37cde063-d68a-48ca-88db-7cf8a26b7b39" (UID: "37cde063-d68a-48ca-88db-7cf8a26b7b39"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:36:29.186265 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.186214    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cde063-d68a-48ca-88db-7cf8a26b7b39-kube-api-access-9llpj" (OuterVolumeSpecName: "kube-api-access-9llpj") pod "37cde063-d68a-48ca-88db-7cf8a26b7b39" (UID: "37cde063-d68a-48ca-88db-7cf8a26b7b39"). InnerVolumeSpecName "kube-api-access-9llpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:36:29.285030 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.284937    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9llpj\" (UniqueName: \"kubernetes.io/projected/37cde063-d68a-48ca-88db-7cf8a26b7b39-kube-api-access-9llpj\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:36:29.285030 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.284969    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37cde063-d68a-48ca-88db-7cf8a26b7b39-cabundle-cert\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:36:29.285030 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.284984    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37cde063-d68a-48ca-88db-7cf8a26b7b39-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:36:29.285030 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.284996    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37cde063-d68a-48ca-88db-7cf8a26b7b39-proxy-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:36:29.973672 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.973637    2578 generic.go:356] "Generic (PLEG): container finished" podID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerID="83147f9d7dddad86388b824a2f46d749dadb7fe77a3f5893d4421dddf042a08c" exitCode=0
Sep 08 13:36:29.973969 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.973706    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"
Sep 08 13:36:29.973969 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.973711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" event={"ID":"37cde063-d68a-48ca-88db-7cf8a26b7b39","Type":"ContainerDied","Data":"83147f9d7dddad86388b824a2f46d749dadb7fe77a3f5893d4421dddf042a08c"}
Sep 08 13:36:29.973969 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.973794    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt" event={"ID":"37cde063-d68a-48ca-88db-7cf8a26b7b39","Type":"ContainerDied","Data":"201833d27aec92d2842f7e80e9fde522b6f5600ab2a7dc9c4b8236c607527b75"}
Sep 08 13:36:29.973969 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.973840    2578 scope.go:117] "RemoveContainer" containerID="83147f9d7dddad86388b824a2f46d749dadb7fe77a3f5893d4421dddf042a08c"
Sep 08 13:36:29.986000 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.985973    2578 scope.go:117] "RemoveContainer" containerID="eaeda957de2a41f78387f295c28f23afaf9c0ec0bc5b6af5c3001b81c2f46d37"
Sep 08 13:36:29.993581 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:29.993562    2578 scope.go:117] "RemoveContainer" containerID="a919f789da0be64172477c87f1918fc26e84827afe1677e48223f02df316622e"
Sep 08 13:36:30.000826 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:30.000807    2578 scope.go:117] "RemoveContainer" containerID="50210e5e179099070e1602a3f9b45e8792b45b6a057a62a0cfaeb44ce882b040"
Sep 08 13:36:30.002957 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:30.002935    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"]
Sep 08 13:36:30.005018 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:30.004999    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-27b1d-predictor-65ccdb9f6c-bskxt"]
Sep 08 13:36:30.008183 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:30.008163    2578 scope.go:117] "RemoveContainer" containerID="83147f9d7dddad86388b824a2f46d749dadb7fe77a3f5893d4421dddf042a08c"
Sep 08 13:36:30.008418 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:36:30.008401    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83147f9d7dddad86388b824a2f46d749dadb7fe77a3f5893d4421dddf042a08c\": container with ID starting with 83147f9d7dddad86388b824a2f46d749dadb7fe77a3f5893d4421dddf042a08c not found: ID does not exist" containerID="83147f9d7dddad86388b824a2f46d749dadb7fe77a3f5893d4421dddf042a08c"
Sep 08 13:36:30.008468 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:30.008427    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83147f9d7dddad86388b824a2f46d749dadb7fe77a3f5893d4421dddf042a08c"} err="failed to get container status \"83147f9d7dddad86388b824a2f46d749dadb7fe77a3f5893d4421dddf042a08c\": rpc error: code = NotFound desc = could not find container \"83147f9d7dddad86388b824a2f46d749dadb7fe77a3f5893d4421dddf042a08c\": container with ID starting with 83147f9d7dddad86388b824a2f46d749dadb7fe77a3f5893d4421dddf042a08c not found: ID does not exist"
Sep 08 13:36:30.008468 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:30.008444    2578 scope.go:117] "RemoveContainer" containerID="eaeda957de2a41f78387f295c28f23afaf9c0ec0bc5b6af5c3001b81c2f46d37"
Sep 08 13:36:30.008681 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:36:30.008667    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaeda957de2a41f78387f295c28f23afaf9c0ec0bc5b6af5c3001b81c2f46d37\": container with ID starting with eaeda957de2a41f78387f295c28f23afaf9c0ec0bc5b6af5c3001b81c2f46d37 not found: ID does not exist" containerID="eaeda957de2a41f78387f295c28f23afaf9c0ec0bc5b6af5c3001b81c2f46d37"
Sep 08 13:36:30.008729 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:30.008684    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaeda957de2a41f78387f295c28f23afaf9c0ec0bc5b6af5c3001b81c2f46d37"} err="failed to get container status \"eaeda957de2a41f78387f295c28f23afaf9c0ec0bc5b6af5c3001b81c2f46d37\": rpc error: code = NotFound desc = could not find container \"eaeda957de2a41f78387f295c28f23afaf9c0ec0bc5b6af5c3001b81c2f46d37\": container with ID starting with eaeda957de2a41f78387f295c28f23afaf9c0ec0bc5b6af5c3001b81c2f46d37 not found: ID does not exist"
Sep 08 13:36:30.008729 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:30.008696    2578 scope.go:117] "RemoveContainer" containerID="a919f789da0be64172477c87f1918fc26e84827afe1677e48223f02df316622e"
Sep 08 13:36:30.008902 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:36:30.008887    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a919f789da0be64172477c87f1918fc26e84827afe1677e48223f02df316622e\": container with ID starting with a919f789da0be64172477c87f1918fc26e84827afe1677e48223f02df316622e not found: ID does not exist" containerID="a919f789da0be64172477c87f1918fc26e84827afe1677e48223f02df316622e"
Sep 08 13:36:30.008943 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:30.008905    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a919f789da0be64172477c87f1918fc26e84827afe1677e48223f02df316622e"} err="failed to get container status \"a919f789da0be64172477c87f1918fc26e84827afe1677e48223f02df316622e\": rpc error: code = NotFound desc = could not find container \"a919f789da0be64172477c87f1918fc26e84827afe1677e48223f02df316622e\": container with ID starting with a919f789da0be64172477c87f1918fc26e84827afe1677e48223f02df316622e not found: ID does not exist"
Sep 08 13:36:30.008943 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:30.008916    2578 scope.go:117] "RemoveContainer" containerID="50210e5e179099070e1602a3f9b45e8792b45b6a057a62a0cfaeb44ce882b040"
Sep 08 13:36:30.009110 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:36:30.009096    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50210e5e179099070e1602a3f9b45e8792b45b6a057a62a0cfaeb44ce882b040\": container with ID starting with 50210e5e179099070e1602a3f9b45e8792b45b6a057a62a0cfaeb44ce882b040 not found: ID does not exist" containerID="50210e5e179099070e1602a3f9b45e8792b45b6a057a62a0cfaeb44ce882b040"
Sep 08 13:36:30.009150 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:30.009113    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50210e5e179099070e1602a3f9b45e8792b45b6a057a62a0cfaeb44ce882b040"} err="failed to get container status \"50210e5e179099070e1602a3f9b45e8792b45b6a057a62a0cfaeb44ce882b040\": rpc error: code = NotFound desc = could not find container \"50210e5e179099070e1602a3f9b45e8792b45b6a057a62a0cfaeb44ce882b040\": container with ID starting with 50210e5e179099070e1602a3f9b45e8792b45b6a057a62a0cfaeb44ce882b040 not found: ID does not exist"
Sep 08 13:36:31.012864 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:36:31.012824    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" path="/var/lib/kubelet/pods/37cde063-d68a-48ca-88db-7cf8a26b7b39/volumes"
Sep 08 13:37:30.425816 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.425755    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"]
Sep 08 13:37:30.426311 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426210    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb474c84-e871-489e-8bc9-6e8cba8adcb5" containerName="kube-rbac-proxy"
Sep 08 13:37:30.426311 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426228    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb474c84-e871-489e-8bc9-6e8cba8adcb5" containerName="kube-rbac-proxy"
Sep 08 13:37:30.426311 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426240    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb474c84-e871-489e-8bc9-6e8cba8adcb5" containerName="kserve-container"
Sep 08 13:37:30.426311 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426249    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb474c84-e871-489e-8bc9-6e8cba8adcb5" containerName="kserve-container"
Sep 08 13:37:30.426311 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426258    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="storage-initializer"
Sep 08 13:37:30.426311 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426267    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="storage-initializer"
Sep 08 13:37:30.426311 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426290    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kube-rbac-proxy"
Sep 08 13:37:30.426311 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426299    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kube-rbac-proxy"
Sep 08 13:37:30.426777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426321    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent"
Sep 08 13:37:30.426777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426329    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent"
Sep 08 13:37:30.426777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426342    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container"
Sep 08 13:37:30.426777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426349    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container"
Sep 08 13:37:30.426777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426442    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kserve-container"
Sep 08 13:37:30.426777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426456    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb474c84-e871-489e-8bc9-6e8cba8adcb5" containerName="kserve-container"
Sep 08 13:37:30.426777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426467    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="kube-rbac-proxy"
Sep 08 13:37:30.426777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426477    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="37cde063-d68a-48ca-88db-7cf8a26b7b39" containerName="agent"
Sep 08 13:37:30.426777 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.426488    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb474c84-e871-489e-8bc9-6e8cba8adcb5" containerName="kube-rbac-proxy"
Sep 08 13:37:30.431092 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.431073    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.433802 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.433775    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-e88a7-predictor-serving-cert\""
Sep 08 13:37:30.433802 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.433794    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-48hwc\""
Sep 08 13:37:30.434025 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.433781    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:37:30.434025 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.433826    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:37:30.434025 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.433850    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:37:30.434863 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.434845    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:37:30.434962 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.434879    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-e88a7-kube-rbac-proxy-sar-config\""
Sep 08 13:37:30.441443 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.441422    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"]
Sep 08 13:37:30.574360 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.574332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kserve-provision-location\") pod \"raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.574515 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.574377    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfdbb\" (UniqueName: \"kubernetes.io/projected/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kube-api-access-pfdbb\") pod \"raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.574515 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.574434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.574515 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.574470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/563636fc-cd17-4456-9cb5-f82fe8abbf2d-cabundle-cert\") pod \"raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.574515 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.574505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/563636fc-cd17-4456-9cb5-f82fe8abbf2d-proxy-tls\") pod \"raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.675482 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.675454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/563636fc-cd17-4456-9cb5-f82fe8abbf2d-proxy-tls\") pod \"raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.675612 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.675514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kserve-provision-location\") pod \"raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.675612 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.675545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfdbb\" (UniqueName: \"kubernetes.io/projected/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kube-api-access-pfdbb\") pod \"raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.675612 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.675576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.675612 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.675602    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/563636fc-cd17-4456-9cb5-f82fe8abbf2d-cabundle-cert\") pod \"raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.675925 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.675880    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kserve-provision-location\") pod \"raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.676362 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.676340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.676411 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.676340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/563636fc-cd17-4456-9cb5-f82fe8abbf2d-cabundle-cert\") pod \"raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.677920 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.677899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/563636fc-cd17-4456-9cb5-f82fe8abbf2d-proxy-tls\") pod \"raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.685230 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.685210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfdbb\" (UniqueName: \"kubernetes.io/projected/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kube-api-access-pfdbb\") pod \"raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:30.742252 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:30.742223    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:31.016223 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:31.016198    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"]
Sep 08 13:37:31.018743 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:37:31.018717    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563636fc_cd17_4456_9cb5_f82fe8abbf2d.slice/crio-c7889731126294d2460fcb979c5b3837ff595fc621c012904ac7960ff142323f WatchSource:0}: Error finding container c7889731126294d2460fcb979c5b3837ff595fc621c012904ac7960ff142323f: Status 404 returned error can't find the container with id c7889731126294d2460fcb979c5b3837ff595fc621c012904ac7960ff142323f
Sep 08 13:37:31.206726 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:31.206691    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" event={"ID":"563636fc-cd17-4456-9cb5-f82fe8abbf2d","Type":"ContainerStarted","Data":"f36ac64a82da95ddb653154b10ad1c2d92733611af0d8bfe121e5b1096c5968f"}
Sep 08 13:37:31.206927 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:31.206748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" event={"ID":"563636fc-cd17-4456-9cb5-f82fe8abbf2d","Type":"ContainerStarted","Data":"c7889731126294d2460fcb979c5b3837ff595fc621c012904ac7960ff142323f"}
Sep 08 13:37:35.225054 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:35.224978    2578 generic.go:356] "Generic (PLEG): container finished" podID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerID="f36ac64a82da95ddb653154b10ad1c2d92733611af0d8bfe121e5b1096c5968f" exitCode=0
Sep 08 13:37:35.225408 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:35.225047    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" event={"ID":"563636fc-cd17-4456-9cb5-f82fe8abbf2d","Type":"ContainerDied","Data":"f36ac64a82da95ddb653154b10ad1c2d92733611af0d8bfe121e5b1096c5968f"}
Sep 08 13:37:36.230849 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:36.230815    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" event={"ID":"563636fc-cd17-4456-9cb5-f82fe8abbf2d","Type":"ContainerStarted","Data":"abd4640fe232a2338a086ddf7f6e7ddae464b166e04997d70b795a270dbc88ee"}
Sep 08 13:37:36.231221 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:36.230876    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" event={"ID":"563636fc-cd17-4456-9cb5-f82fe8abbf2d","Type":"ContainerStarted","Data":"1336b80acb576fbf207fb6a0331292e40c0b57085f4fb9da87b0c072d6ed2e63"}
Sep 08 13:37:36.231221 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:36.231069    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:36.252646 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:36.252585    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" podStartSLOduration=2.419033011 podStartE2EDuration="6.252571341s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.83353833s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:37:36.250718237 +0000 UTC m=+1273.850815865" watchObservedRunningTime="2026-09-08 13:37:36.252571341 +0000 UTC m=+1273.852668967"
Sep 08 13:37:37.240741 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:37.240708    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:37.241984 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:37.241954    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:37:38.244770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:38.244732    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:37:43.249613 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:43.249585    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:37:43.250179 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:43.250152    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:37:53.250247 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:37:53.250205    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:38:03.250770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:03.250728    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:38:13.250666 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:13.250609    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:38:23.250634 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:23.250579    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:38:33.250190 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:33.250150    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:38:43.250503 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:43.250462    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:38:44.366254 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.366219    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"]
Sep 08 13:38:44.370484 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.370455    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:44.373562 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.373537    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 08 13:38:44.373716 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.373612    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 08 13:38:44.384050 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.384027    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"]
Sep 08 13:38:44.401677 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.401649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg7hz\" (UniqueName: \"kubernetes.io/projected/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kube-api-access-sg7hz\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:44.401809 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.401704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:44.401809 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.401733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:44.401809 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.401754    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:44.401809 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.401789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:44.440756 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.440726    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"]
Sep 08 13:38:44.444182 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.444162    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.446778 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.446755    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 08 13:38:44.456304 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.456283    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"]
Sep 08 13:38:44.502965 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.502931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:44.503133 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.502977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d38680b5-dc8a-44dc-af18-1e2f55fff934-proxy-tls\") pod \"isvc-canary-create-predictor-8488566dd5-btw8t\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.503133 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.503014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:44.503133 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.503045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:44.503133 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.503070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d38680b5-dc8a-44dc-af18-1e2f55fff934-cabundle-cert\") pod \"isvc-canary-create-predictor-8488566dd5-btw8t\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.503133 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.503125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d38680b5-dc8a-44dc-af18-1e2f55fff934-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8488566dd5-btw8t\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.503404 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.503180    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d38680b5-dc8a-44dc-af18-1e2f55fff934-kserve-provision-location\") pod \"isvc-canary-create-predictor-8488566dd5-btw8t\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.503404 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.503208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg7hz\" (UniqueName: \"kubernetes.io/projected/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kube-api-access-sg7hz\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:44.503404 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.503246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ltr6\" (UniqueName: \"kubernetes.io/projected/d38680b5-dc8a-44dc-af18-1e2f55fff934-kube-api-access-4ltr6\") pod \"isvc-canary-create-predictor-8488566dd5-btw8t\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.503404 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.503301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:44.503610 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:38:44.503413    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 08 13:38:44.503610 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:38:44.503464    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-proxy-tls podName:2358e1b9-dd70-4b1f-a360-3da2cc3ded53 nodeName:}" failed. No retries permitted until 2026-09-08 13:38:45.003446111 +0000 UTC m=+1342.603543720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-proxy-tls") pod "isvc-canary-create-v2-predictor-7659888cb8-ktcnp" (UID: "2358e1b9-dd70-4b1f-a360-3da2cc3ded53") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 08 13:38:44.503610 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.503491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:44.503807 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.503715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:44.503807 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.503715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:44.511910 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.511892    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg7hz\" (UniqueName: \"kubernetes.io/projected/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kube-api-access-sg7hz\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:44.604333 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.604306    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d38680b5-dc8a-44dc-af18-1e2f55fff934-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8488566dd5-btw8t\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.604485 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.604354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d38680b5-dc8a-44dc-af18-1e2f55fff934-kserve-provision-location\") pod \"isvc-canary-create-predictor-8488566dd5-btw8t\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.604485 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.604397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ltr6\" (UniqueName: \"kubernetes.io/projected/d38680b5-dc8a-44dc-af18-1e2f55fff934-kube-api-access-4ltr6\") pod \"isvc-canary-create-predictor-8488566dd5-btw8t\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.604485 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.604463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d38680b5-dc8a-44dc-af18-1e2f55fff934-proxy-tls\") pod \"isvc-canary-create-predictor-8488566dd5-btw8t\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.604724 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.604500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d38680b5-dc8a-44dc-af18-1e2f55fff934-cabundle-cert\") pod \"isvc-canary-create-predictor-8488566dd5-btw8t\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.604907 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.604878    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d38680b5-dc8a-44dc-af18-1e2f55fff934-kserve-provision-location\") pod \"isvc-canary-create-predictor-8488566dd5-btw8t\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.605157 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.605134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d38680b5-dc8a-44dc-af18-1e2f55fff934-cabundle-cert\") pod \"isvc-canary-create-predictor-8488566dd5-btw8t\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.605239 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.605164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d38680b5-dc8a-44dc-af18-1e2f55fff934-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8488566dd5-btw8t\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.606882 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.606864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d38680b5-dc8a-44dc-af18-1e2f55fff934-proxy-tls\") pod \"isvc-canary-create-predictor-8488566dd5-btw8t\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.617715 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.617664    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ltr6\" (UniqueName: \"kubernetes.io/projected/d38680b5-dc8a-44dc-af18-1e2f55fff934-kube-api-access-4ltr6\") pod \"isvc-canary-create-predictor-8488566dd5-btw8t\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:44.754722 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:44.754682    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:45.008767 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:45.008727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:45.011448 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:45.011425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-proxy-tls\") pod \"isvc-canary-create-v2-predictor-7659888cb8-ktcnp\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:45.034817 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:45.034795    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"]
Sep 08 13:38:45.037340 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:38:45.037312    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38680b5_dc8a_44dc_af18_1e2f55fff934.slice/crio-6845c05866da314a4f07ee16d57644d0e53c4208a9e7a06f503157c59e1fed58 WatchSource:0}: Error finding container 6845c05866da314a4f07ee16d57644d0e53c4208a9e7a06f503157c59e1fed58: Status 404 returned error can't find the container with id 6845c05866da314a4f07ee16d57644d0e53c4208a9e7a06f503157c59e1fed58
Sep 08 13:38:45.281255 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:45.281135    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:45.529024 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:45.528991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" event={"ID":"d38680b5-dc8a-44dc-af18-1e2f55fff934","Type":"ContainerStarted","Data":"f8f7bcba9c0723634891305a1b53e58244d819131908baf1de328a9a0bb2f7ed"}
Sep 08 13:38:45.529392 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:45.529072    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" event={"ID":"d38680b5-dc8a-44dc-af18-1e2f55fff934","Type":"ContainerStarted","Data":"6845c05866da314a4f07ee16d57644d0e53c4208a9e7a06f503157c59e1fed58"}
Sep 08 13:38:45.555007 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:45.554977    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"]
Sep 08 13:38:45.555971 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:38:45.555948    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2358e1b9_dd70_4b1f_a360_3da2cc3ded53.slice/crio-2a1330b89433b164ab73c3766b8f7d9647e55653fa1fb7d9df14675477572122 WatchSource:0}: Error finding container 2a1330b89433b164ab73c3766b8f7d9647e55653fa1fb7d9df14675477572122: Status 404 returned error can't find the container with id 2a1330b89433b164ab73c3766b8f7d9647e55653fa1fb7d9df14675477572122
Sep 08 13:38:46.534467 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:46.534434    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" event={"ID":"2358e1b9-dd70-4b1f-a360-3da2cc3ded53","Type":"ContainerStarted","Data":"b9a668abed899b97c7b50244d789ff02f61b3a1f67629647128467a56b76503e"}
Sep 08 13:38:46.534867 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:46.534527    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" event={"ID":"2358e1b9-dd70-4b1f-a360-3da2cc3ded53","Type":"ContainerStarted","Data":"2a1330b89433b164ab73c3766b8f7d9647e55653fa1fb7d9df14675477572122"}
Sep 08 13:38:49.547368 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:49.547327    2578 generic.go:356] "Generic (PLEG): container finished" podID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerID="f8f7bcba9c0723634891305a1b53e58244d819131908baf1de328a9a0bb2f7ed" exitCode=0
Sep 08 13:38:49.547823 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:49.547401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" event={"ID":"d38680b5-dc8a-44dc-af18-1e2f55fff934","Type":"ContainerDied","Data":"f8f7bcba9c0723634891305a1b53e58244d819131908baf1de328a9a0bb2f7ed"}
Sep 08 13:38:49.549011 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:49.548992    2578 generic.go:356] "Generic (PLEG): container finished" podID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerID="b9a668abed899b97c7b50244d789ff02f61b3a1f67629647128467a56b76503e" exitCode=0
Sep 08 13:38:49.549092 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:49.549048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" event={"ID":"2358e1b9-dd70-4b1f-a360-3da2cc3ded53","Type":"ContainerDied","Data":"b9a668abed899b97c7b50244d789ff02f61b3a1f67629647128467a56b76503e"}
Sep 08 13:38:50.561068 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:50.561028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" event={"ID":"d38680b5-dc8a-44dc-af18-1e2f55fff934","Type":"ContainerStarted","Data":"f56f02b0c225586d53d763b600adef9191e9b85bc60c59580243d74becd561e3"}
Sep 08 13:38:50.561544 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:50.561116    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" event={"ID":"d38680b5-dc8a-44dc-af18-1e2f55fff934","Type":"ContainerStarted","Data":"e20c632c8186a4fc8afd7f5f8fe4bc7639398b94a803b2c47b4415e4e028a602"}
Sep 08 13:38:50.561544 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:50.561293    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:50.563315 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:50.563292    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" event={"ID":"2358e1b9-dd70-4b1f-a360-3da2cc3ded53","Type":"ContainerStarted","Data":"37a9c4154a52b8191542f1f45400ad94938688912d13c40ace92767cb44b84e0"}
Sep 08 13:38:50.563427 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:50.563338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" event={"ID":"2358e1b9-dd70-4b1f-a360-3da2cc3ded53","Type":"ContainerStarted","Data":"b269e69ef17c9041b4941c42dcc67c25260ccdbdaf3b0b47532bb07fc94b0a14"}
Sep 08 13:38:50.563522 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:50.563504    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:50.563580 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:50.563546    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:50.564943 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:50.564918    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:38:50.584137 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:50.584093    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" podStartSLOduration=2.8930199979999998 podStartE2EDuration="6.584077443s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.691057445s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:38:50.583271759 +0000 UTC m=+1348.183369384" watchObservedRunningTime="2026-09-08 13:38:50.584077443 +0000 UTC m=+1348.184175093"
Sep 08 13:38:50.606780 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:50.606739    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" podStartSLOduration=2.868999245 podStartE2EDuration="6.60672512s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.737725875s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:38:50.605023767 +0000 UTC m=+1348.205121422" watchObservedRunningTime="2026-09-08 13:38:50.60672512 +0000 UTC m=+1348.206822745"
Sep 08 13:38:51.567669 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:51.567603    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:38:51.568143 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:51.567832    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:51.569225 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:51.569191    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:38:52.573266 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:52.573217    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:38:53.250717 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:53.250687    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:38:56.573378 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:56.573345    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:38:56.580667 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:56.574226    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:38:57.578670 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:57.578636    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:38:57.579263 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:38:57.579238    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:39:00.471366 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:00.471324    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"]
Sep 08 13:39:00.471828 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:00.471693    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kserve-container" containerID="cri-o://1336b80acb576fbf207fb6a0331292e40c0b57085f4fb9da87b0c072d6ed2e63" gracePeriod=30
Sep 08 13:39:00.471828 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:00.471721    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kube-rbac-proxy" containerID="cri-o://abd4640fe232a2338a086ddf7f6e7ddae464b166e04997d70b795a270dbc88ee" gracePeriod=30
Sep 08 13:39:00.610800 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:00.610762    2578 generic.go:356] "Generic (PLEG): container finished" podID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerID="abd4640fe232a2338a086ddf7f6e7ddae464b166e04997d70b795a270dbc88ee" exitCode=2
Sep 08 13:39:00.610951 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:00.610827    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" event={"ID":"563636fc-cd17-4456-9cb5-f82fe8abbf2d","Type":"ContainerDied","Data":"abd4640fe232a2338a086ddf7f6e7ddae464b166e04997d70b795a270dbc88ee"}
Sep 08 13:39:03.245724 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:03.245680    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 08 13:39:03.250157 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:03.250119    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 08 13:39:05.120154 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.120128    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:39:05.186034 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.185992    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/563636fc-cd17-4456-9cb5-f82fe8abbf2d-cabundle-cert\") pod \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") "
Sep 08 13:39:05.186200 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.186057    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/563636fc-cd17-4456-9cb5-f82fe8abbf2d-proxy-tls\") pod \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") "
Sep 08 13:39:05.186200 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.186103    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kserve-provision-location\") pod \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") "
Sep 08 13:39:05.186200 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.186127    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kube-rbac-proxy-sar-config\") pod \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") "
Sep 08 13:39:05.186200 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.186184    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfdbb\" (UniqueName: \"kubernetes.io/projected/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kube-api-access-pfdbb\") pod \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\" (UID: \"563636fc-cd17-4456-9cb5-f82fe8abbf2d\") "
Sep 08 13:39:05.186476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.186443    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "563636fc-cd17-4456-9cb5-f82fe8abbf2d" (UID: "563636fc-cd17-4456-9cb5-f82fe8abbf2d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:39:05.186544 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.186486    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/563636fc-cd17-4456-9cb5-f82fe8abbf2d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "563636fc-cd17-4456-9cb5-f82fe8abbf2d" (UID: "563636fc-cd17-4456-9cb5-f82fe8abbf2d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:39:05.186544 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.186529    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "563636fc-cd17-4456-9cb5-f82fe8abbf2d" (UID: "563636fc-cd17-4456-9cb5-f82fe8abbf2d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:39:05.188114 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.188094    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563636fc-cd17-4456-9cb5-f82fe8abbf2d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "563636fc-cd17-4456-9cb5-f82fe8abbf2d" (UID: "563636fc-cd17-4456-9cb5-f82fe8abbf2d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:39:05.188229 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.188208    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kube-api-access-pfdbb" (OuterVolumeSpecName: "kube-api-access-pfdbb") pod "563636fc-cd17-4456-9cb5-f82fe8abbf2d" (UID: "563636fc-cd17-4456-9cb5-f82fe8abbf2d"). InnerVolumeSpecName "kube-api-access-pfdbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:39:05.286992 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.286916    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/563636fc-cd17-4456-9cb5-f82fe8abbf2d-cabundle-cert\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:39:05.286992 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.286944    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/563636fc-cd17-4456-9cb5-f82fe8abbf2d-proxy-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:39:05.286992 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.286954    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kserve-provision-location\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:39:05.286992 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.286963    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:39:05.286992 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.286971    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfdbb\" (UniqueName: \"kubernetes.io/projected/563636fc-cd17-4456-9cb5-f82fe8abbf2d-kube-api-access-pfdbb\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:39:05.632641 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.632531    2578 generic.go:356] "Generic (PLEG): container finished" podID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerID="1336b80acb576fbf207fb6a0331292e40c0b57085f4fb9da87b0c072d6ed2e63" exitCode=0
Sep 08 13:39:05.632641 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.632573    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" event={"ID":"563636fc-cd17-4456-9cb5-f82fe8abbf2d","Type":"ContainerDied","Data":"1336b80acb576fbf207fb6a0331292e40c0b57085f4fb9da87b0c072d6ed2e63"}
Sep 08 13:39:05.632641 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.632604    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"
Sep 08 13:39:05.632641 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.632639    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q" event={"ID":"563636fc-cd17-4456-9cb5-f82fe8abbf2d","Type":"ContainerDied","Data":"c7889731126294d2460fcb979c5b3837ff595fc621c012904ac7960ff142323f"}
Sep 08 13:39:05.632967 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.632668    2578 scope.go:117] "RemoveContainer" containerID="abd4640fe232a2338a086ddf7f6e7ddae464b166e04997d70b795a270dbc88ee"
Sep 08 13:39:05.641866 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.641846    2578 scope.go:117] "RemoveContainer" containerID="1336b80acb576fbf207fb6a0331292e40c0b57085f4fb9da87b0c072d6ed2e63"
Sep 08 13:39:05.649262 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.649241    2578 scope.go:117] "RemoveContainer" containerID="f36ac64a82da95ddb653154b10ad1c2d92733611af0d8bfe121e5b1096c5968f"
Sep 08 13:39:05.656285 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.656262    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"]
Sep 08 13:39:05.656954 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.656938    2578 scope.go:117] "RemoveContainer" containerID="abd4640fe232a2338a086ddf7f6e7ddae464b166e04997d70b795a270dbc88ee"
Sep 08 13:39:05.657229 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:39:05.657211    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd4640fe232a2338a086ddf7f6e7ddae464b166e04997d70b795a270dbc88ee\": container with ID starting with abd4640fe232a2338a086ddf7f6e7ddae464b166e04997d70b795a270dbc88ee not found: ID does not exist" containerID="abd4640fe232a2338a086ddf7f6e7ddae464b166e04997d70b795a270dbc88ee"
Sep 08 13:39:05.657282 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.657238    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd4640fe232a2338a086ddf7f6e7ddae464b166e04997d70b795a270dbc88ee"} err="failed to get container status \"abd4640fe232a2338a086ddf7f6e7ddae464b166e04997d70b795a270dbc88ee\": rpc error: code = NotFound desc = could not find container \"abd4640fe232a2338a086ddf7f6e7ddae464b166e04997d70b795a270dbc88ee\": container with ID starting with abd4640fe232a2338a086ddf7f6e7ddae464b166e04997d70b795a270dbc88ee not found: ID does not exist"
Sep 08 13:39:05.657282 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.657256    2578 scope.go:117] "RemoveContainer" containerID="1336b80acb576fbf207fb6a0331292e40c0b57085f4fb9da87b0c072d6ed2e63"
Sep 08 13:39:05.657479 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:39:05.657460    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1336b80acb576fbf207fb6a0331292e40c0b57085f4fb9da87b0c072d6ed2e63\": container with ID starting with 1336b80acb576fbf207fb6a0331292e40c0b57085f4fb9da87b0c072d6ed2e63 not found: ID does not exist" containerID="1336b80acb576fbf207fb6a0331292e40c0b57085f4fb9da87b0c072d6ed2e63"
Sep 08 13:39:05.657542 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.657489    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1336b80acb576fbf207fb6a0331292e40c0b57085f4fb9da87b0c072d6ed2e63"} err="failed to get container status \"1336b80acb576fbf207fb6a0331292e40c0b57085f4fb9da87b0c072d6ed2e63\": rpc error: code = NotFound desc = could not find container \"1336b80acb576fbf207fb6a0331292e40c0b57085f4fb9da87b0c072d6ed2e63\": container with ID starting with 1336b80acb576fbf207fb6a0331292e40c0b57085f4fb9da87b0c072d6ed2e63 not found: ID does not exist"
Sep 08 13:39:05.657542 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.657513    2578 scope.go:117] "RemoveContainer" containerID="f36ac64a82da95ddb653154b10ad1c2d92733611af0d8bfe121e5b1096c5968f"
Sep 08 13:39:05.657779 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:39:05.657760    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36ac64a82da95ddb653154b10ad1c2d92733611af0d8bfe121e5b1096c5968f\": container with ID starting with f36ac64a82da95ddb653154b10ad1c2d92733611af0d8bfe121e5b1096c5968f not found: ID does not exist" containerID="f36ac64a82da95ddb653154b10ad1c2d92733611af0d8bfe121e5b1096c5968f"
Sep 08 13:39:05.657847 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.657786    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36ac64a82da95ddb653154b10ad1c2d92733611af0d8bfe121e5b1096c5968f"} err="failed to get container status \"f36ac64a82da95ddb653154b10ad1c2d92733611af0d8bfe121e5b1096c5968f\": rpc error: code = NotFound desc = could not find container \"f36ac64a82da95ddb653154b10ad1c2d92733611af0d8bfe121e5b1096c5968f\": container with ID starting with f36ac64a82da95ddb653154b10ad1c2d92733611af0d8bfe121e5b1096c5968f not found: ID does not exist"
Sep 08 13:39:05.660476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:05.660454    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-e88a7-predictor-5d7f68d48-nks6q"]
Sep 08 13:39:06.574580 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:06.574545    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:39:07.013703 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:07.013665    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" path="/var/lib/kubelet/pods/563636fc-cd17-4456-9cb5-f82fe8abbf2d/volumes"
Sep 08 13:39:07.583374 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:07.583323    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:39:16.574692 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:16.574650    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:39:17.579173 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:17.579127    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:39:26.573931 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:26.573873    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:39:27.579425 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:27.579381    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:39:36.574268 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:36.574222    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:39:37.579674 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:37.579631    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:39:46.574763 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:46.574715    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:39:47.580038 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:47.579994    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:39:56.574553 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:56.574508    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:39:57.579450 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:57.579407    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:39:59.008716 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:39:59.008675    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 08 13:40:02.008255 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:02.008212    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 08 13:40:09.012158 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:09.012127    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:40:12.008779 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:12.008749    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:40:12.197247 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:12.197217    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"]
Sep 08 13:40:12.197514 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:12.197473    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kserve-container" containerID="cri-o://e20c632c8186a4fc8afd7f5f8fe4bc7639398b94a803b2c47b4415e4e028a602" gracePeriod=30
Sep 08 13:40:12.197576 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:12.197513    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kube-rbac-proxy" containerID="cri-o://f56f02b0c225586d53d763b600adef9191e9b85bc60c59580243d74becd561e3" gracePeriod=30
Sep 08 13:40:12.573595 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:12.573479    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 08 13:40:12.913172 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:12.913137    2578 generic.go:356] "Generic (PLEG): container finished" podID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerID="f56f02b0c225586d53d763b600adef9191e9b85bc60c59580243d74becd561e3" exitCode=2
Sep 08 13:40:12.913335 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:12.913175    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" event={"ID":"d38680b5-dc8a-44dc-af18-1e2f55fff934","Type":"ContainerDied","Data":"f56f02b0c225586d53d763b600adef9191e9b85bc60c59580243d74becd561e3"}
Sep 08 13:40:14.464341 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:14.464303    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"]
Sep 08 13:40:14.464736 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:14.464608    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kserve-container" containerID="cri-o://b269e69ef17c9041b4941c42dcc67c25260ccdbdaf3b0b47532bb07fc94b0a14" gracePeriod=30
Sep 08 13:40:14.464736 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:14.464703    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kube-rbac-proxy" containerID="cri-o://37a9c4154a52b8191542f1f45400ad94938688912d13c40ace92767cb44b84e0" gracePeriod=30
Sep 08 13:40:14.923391 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:14.923358    2578 generic.go:356] "Generic (PLEG): container finished" podID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerID="37a9c4154a52b8191542f1f45400ad94938688912d13c40ace92767cb44b84e0" exitCode=2
Sep 08 13:40:14.923560 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:14.923434    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" event={"ID":"2358e1b9-dd70-4b1f-a360-3da2cc3ded53","Type":"ContainerDied","Data":"37a9c4154a52b8191542f1f45400ad94938688912d13c40ace92767cb44b84e0"}
Sep 08 13:40:16.568877 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:16.568825    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 08 13:40:16.934607 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:16.934578    2578 generic.go:356] "Generic (PLEG): container finished" podID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerID="e20c632c8186a4fc8afd7f5f8fe4bc7639398b94a803b2c47b4415e4e028a602" exitCode=0
Sep 08 13:40:16.934754 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:16.934656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" event={"ID":"d38680b5-dc8a-44dc-af18-1e2f55fff934","Type":"ContainerDied","Data":"e20c632c8186a4fc8afd7f5f8fe4bc7639398b94a803b2c47b4415e4e028a602"}
Sep 08 13:40:16.934754 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:16.934745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t" event={"ID":"d38680b5-dc8a-44dc-af18-1e2f55fff934","Type":"ContainerDied","Data":"6845c05866da314a4f07ee16d57644d0e53c4208a9e7a06f503157c59e1fed58"}
Sep 08 13:40:16.934833 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:16.934767    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6845c05866da314a4f07ee16d57644d0e53c4208a9e7a06f503157c59e1fed58"
Sep 08 13:40:16.939610 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:16.939589    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:40:17.000141 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.000115    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d38680b5-dc8a-44dc-af18-1e2f55fff934-kube-rbac-proxy-sar-config\") pod \"d38680b5-dc8a-44dc-af18-1e2f55fff934\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") "
Sep 08 13:40:17.000305 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.000157    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d38680b5-dc8a-44dc-af18-1e2f55fff934-cabundle-cert\") pod \"d38680b5-dc8a-44dc-af18-1e2f55fff934\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") "
Sep 08 13:40:17.000305 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.000238    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d38680b5-dc8a-44dc-af18-1e2f55fff934-kserve-provision-location\") pod \"d38680b5-dc8a-44dc-af18-1e2f55fff934\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") "
Sep 08 13:40:17.000305 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.000266    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ltr6\" (UniqueName: \"kubernetes.io/projected/d38680b5-dc8a-44dc-af18-1e2f55fff934-kube-api-access-4ltr6\") pod \"d38680b5-dc8a-44dc-af18-1e2f55fff934\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") "
Sep 08 13:40:17.000468 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.000310    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d38680b5-dc8a-44dc-af18-1e2f55fff934-proxy-tls\") pod \"d38680b5-dc8a-44dc-af18-1e2f55fff934\" (UID: \"d38680b5-dc8a-44dc-af18-1e2f55fff934\") "
Sep 08 13:40:17.000517 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.000475    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38680b5-dc8a-44dc-af18-1e2f55fff934-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d38680b5-dc8a-44dc-af18-1e2f55fff934" (UID: "d38680b5-dc8a-44dc-af18-1e2f55fff934"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:40:17.000590 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.000547    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d38680b5-dc8a-44dc-af18-1e2f55fff934-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d38680b5-dc8a-44dc-af18-1e2f55fff934" (UID: "d38680b5-dc8a-44dc-af18-1e2f55fff934"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:40:17.000590 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.000573    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38680b5-dc8a-44dc-af18-1e2f55fff934-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d38680b5-dc8a-44dc-af18-1e2f55fff934" (UID: "d38680b5-dc8a-44dc-af18-1e2f55fff934"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:40:17.000769 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.000657    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d38680b5-dc8a-44dc-af18-1e2f55fff934-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:40:17.002447 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.002418    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38680b5-dc8a-44dc-af18-1e2f55fff934-kube-api-access-4ltr6" (OuterVolumeSpecName: "kube-api-access-4ltr6") pod "d38680b5-dc8a-44dc-af18-1e2f55fff934" (UID: "d38680b5-dc8a-44dc-af18-1e2f55fff934"). InnerVolumeSpecName "kube-api-access-4ltr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:40:17.002533 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.002447    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d38680b5-dc8a-44dc-af18-1e2f55fff934-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d38680b5-dc8a-44dc-af18-1e2f55fff934" (UID: "d38680b5-dc8a-44dc-af18-1e2f55fff934"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:40:17.101340 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.101311    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d38680b5-dc8a-44dc-af18-1e2f55fff934-cabundle-cert\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:40:17.101340 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.101337    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d38680b5-dc8a-44dc-af18-1e2f55fff934-kserve-provision-location\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:40:17.101496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.101346    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ltr6\" (UniqueName: \"kubernetes.io/projected/d38680b5-dc8a-44dc-af18-1e2f55fff934-kube-api-access-4ltr6\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:40:17.101496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.101355    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d38680b5-dc8a-44dc-af18-1e2f55fff934-proxy-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:40:17.938705 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.938677    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"
Sep 08 13:40:17.962630 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.962589    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"]
Sep 08 13:40:17.968462 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:17.968439    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-btw8t"]
Sep 08 13:40:19.013112 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.013071    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" path="/var/lib/kubelet/pods/d38680b5-dc8a-44dc-af18-1e2f55fff934/volumes"
Sep 08 13:40:19.212280 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.212254    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:40:19.219382 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.219364    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kserve-provision-location\") pod \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") "
Sep 08 13:40:19.219457 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.219399    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg7hz\" (UniqueName: \"kubernetes.io/projected/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kube-api-access-sg7hz\") pod \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") "
Sep 08 13:40:19.219457 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.219423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kube-rbac-proxy-sar-config\") pod \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") "
Sep 08 13:40:19.219567 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.219470    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-cabundle-cert\") pod \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") "
Sep 08 13:40:19.219567 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.219553    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-proxy-tls\") pod \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\" (UID: \"2358e1b9-dd70-4b1f-a360-3da2cc3ded53\") "
Sep 08 13:40:19.219729 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.219704    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2358e1b9-dd70-4b1f-a360-3da2cc3ded53" (UID: "2358e1b9-dd70-4b1f-a360-3da2cc3ded53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:40:19.219818 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.219800    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kserve-provision-location\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:40:19.219818 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.219810    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2358e1b9-dd70-4b1f-a360-3da2cc3ded53" (UID: "2358e1b9-dd70-4b1f-a360-3da2cc3ded53"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:40:19.219933 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.219899    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2358e1b9-dd70-4b1f-a360-3da2cc3ded53" (UID: "2358e1b9-dd70-4b1f-a360-3da2cc3ded53"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:40:19.221488 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.221434    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kube-api-access-sg7hz" (OuterVolumeSpecName: "kube-api-access-sg7hz") pod "2358e1b9-dd70-4b1f-a360-3da2cc3ded53" (UID: "2358e1b9-dd70-4b1f-a360-3da2cc3ded53"). InnerVolumeSpecName "kube-api-access-sg7hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:40:19.221596 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.221523    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2358e1b9-dd70-4b1f-a360-3da2cc3ded53" (UID: "2358e1b9-dd70-4b1f-a360-3da2cc3ded53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:40:19.321051 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.321015    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg7hz\" (UniqueName: \"kubernetes.io/projected/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kube-api-access-sg7hz\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:40:19.321051 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.321044    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:40:19.321051 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.321055    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-cabundle-cert\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:40:19.321307 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.321064    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2358e1b9-dd70-4b1f-a360-3da2cc3ded53-proxy-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:40:19.733884 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.733846    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"]
Sep 08 13:40:19.734288 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734275    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kserve-container"
Sep 08 13:40:19.734341 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734292    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kserve-container"
Sep 08 13:40:19.734341 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734305    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kube-rbac-proxy"
Sep 08 13:40:19.734341 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734313    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kube-rbac-proxy"
Sep 08 13:40:19.734341 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734327    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kserve-container"
Sep 08 13:40:19.734341 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734338    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kserve-container"
Sep 08 13:40:19.734492 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734351    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="storage-initializer"
Sep 08 13:40:19.734492 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734362    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="storage-initializer"
Sep 08 13:40:19.734492 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734376    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="storage-initializer"
Sep 08 13:40:19.734492 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734384    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="storage-initializer"
Sep 08 13:40:19.734492 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734394    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kube-rbac-proxy"
Sep 08 13:40:19.734492 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734402    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kube-rbac-proxy"
Sep 08 13:40:19.734492 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734412    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="storage-initializer"
Sep 08 13:40:19.734492 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734420    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="storage-initializer"
Sep 08 13:40:19.734492 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734435    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kube-rbac-proxy"
Sep 08 13:40:19.734492 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734443    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kube-rbac-proxy"
Sep 08 13:40:19.734492 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734458    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kserve-container"
Sep 08 13:40:19.734492 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734466    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kserve-container"
Sep 08 13:40:19.734892 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734530    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kube-rbac-proxy"
Sep 08 13:40:19.734892 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734542    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerName="kserve-container"
Sep 08 13:40:19.734892 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734553    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kserve-container"
Sep 08 13:40:19.734892 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734564    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kserve-container"
Sep 08 13:40:19.734892 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734574    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="563636fc-cd17-4456-9cb5-f82fe8abbf2d" containerName="kube-rbac-proxy"
Sep 08 13:40:19.734892 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.734583    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d38680b5-dc8a-44dc-af18-1e2f55fff934" containerName="kube-rbac-proxy"
Sep 08 13:40:19.738211 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.738184    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:19.740957 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.740932    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 08 13:40:19.741093 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.740939    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 08 13:40:19.754093 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.754071    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"]
Sep 08 13:40:19.825410 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.825384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3b8372b-5073-41cc-8090-b7a72ad77146-proxy-tls\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:19.825575 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.825422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psbkf\" (UniqueName: \"kubernetes.io/projected/b3b8372b-5073-41cc-8090-b7a72ad77146-kube-api-access-psbkf\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:19.825575 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.825449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3b8372b-5073-41cc-8090-b7a72ad77146-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:19.825575 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.825531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3b8372b-5073-41cc-8090-b7a72ad77146-kserve-provision-location\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:19.825764 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.825601    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3b8372b-5073-41cc-8090-b7a72ad77146-cabundle-cert\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:19.926521 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.926485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psbkf\" (UniqueName: \"kubernetes.io/projected/b3b8372b-5073-41cc-8090-b7a72ad77146-kube-api-access-psbkf\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:19.926709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.926528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3b8372b-5073-41cc-8090-b7a72ad77146-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:19.926709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.926545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3b8372b-5073-41cc-8090-b7a72ad77146-kserve-provision-location\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:19.926709 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.926678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3b8372b-5073-41cc-8090-b7a72ad77146-cabundle-cert\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:19.926961 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.926775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3b8372b-5073-41cc-8090-b7a72ad77146-proxy-tls\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:19.926961 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:40:19.926931    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 08 13:40:19.927065 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:40:19.926991    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3b8372b-5073-41cc-8090-b7a72ad77146-proxy-tls podName:b3b8372b-5073-41cc-8090-b7a72ad77146 nodeName:}" failed. No retries permitted until 2026-09-08 13:40:20.426970378 +0000 UTC m=+1438.027067987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b3b8372b-5073-41cc-8090-b7a72ad77146-proxy-tls") pod "isvc-canary-promote-predictor-64849fbd89-m42c6" (UID: "b3b8372b-5073-41cc-8090-b7a72ad77146") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 08 13:40:19.927065 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.927001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3b8372b-5073-41cc-8090-b7a72ad77146-kserve-provision-location\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:19.927323 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.927303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3b8372b-5073-41cc-8090-b7a72ad77146-cabundle-cert\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:19.927368 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.927327    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3b8372b-5073-41cc-8090-b7a72ad77146-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:19.945486 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.945454    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psbkf\" (UniqueName: \"kubernetes.io/projected/b3b8372b-5073-41cc-8090-b7a72ad77146-kube-api-access-psbkf\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:19.948897 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.948871    2578 generic.go:356] "Generic (PLEG): container finished" podID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" containerID="b269e69ef17c9041b4941c42dcc67c25260ccdbdaf3b0b47532bb07fc94b0a14" exitCode=0
Sep 08 13:40:19.949008 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.948908    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" event={"ID":"2358e1b9-dd70-4b1f-a360-3da2cc3ded53","Type":"ContainerDied","Data":"b269e69ef17c9041b4941c42dcc67c25260ccdbdaf3b0b47532bb07fc94b0a14"}
Sep 08 13:40:19.949008 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.948927    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"
Sep 08 13:40:19.949008 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.948973    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp" event={"ID":"2358e1b9-dd70-4b1f-a360-3da2cc3ded53","Type":"ContainerDied","Data":"2a1330b89433b164ab73c3766b8f7d9647e55653fa1fb7d9df14675477572122"}
Sep 08 13:40:19.949170 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.949013    2578 scope.go:117] "RemoveContainer" containerID="37a9c4154a52b8191542f1f45400ad94938688912d13c40ace92767cb44b84e0"
Sep 08 13:40:19.957914 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.957894    2578 scope.go:117] "RemoveContainer" containerID="b269e69ef17c9041b4941c42dcc67c25260ccdbdaf3b0b47532bb07fc94b0a14"
Sep 08 13:40:19.965921 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.965905    2578 scope.go:117] "RemoveContainer" containerID="b9a668abed899b97c7b50244d789ff02f61b3a1f67629647128467a56b76503e"
Sep 08 13:40:19.973534 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.973433    2578 scope.go:117] "RemoveContainer" containerID="37a9c4154a52b8191542f1f45400ad94938688912d13c40ace92767cb44b84e0"
Sep 08 13:40:19.973796 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:40:19.973775    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a9c4154a52b8191542f1f45400ad94938688912d13c40ace92767cb44b84e0\": container with ID starting with 37a9c4154a52b8191542f1f45400ad94938688912d13c40ace92767cb44b84e0 not found: ID does not exist" containerID="37a9c4154a52b8191542f1f45400ad94938688912d13c40ace92767cb44b84e0"
Sep 08 13:40:19.973897 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.973802    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a9c4154a52b8191542f1f45400ad94938688912d13c40ace92767cb44b84e0"} err="failed to get container status \"37a9c4154a52b8191542f1f45400ad94938688912d13c40ace92767cb44b84e0\": rpc error: code = NotFound desc = could not find container \"37a9c4154a52b8191542f1f45400ad94938688912d13c40ace92767cb44b84e0\": container with ID starting with 37a9c4154a52b8191542f1f45400ad94938688912d13c40ace92767cb44b84e0 not found: ID does not exist"
Sep 08 13:40:19.973897 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.973820    2578 scope.go:117] "RemoveContainer" containerID="b269e69ef17c9041b4941c42dcc67c25260ccdbdaf3b0b47532bb07fc94b0a14"
Sep 08 13:40:19.974180 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:40:19.974148    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b269e69ef17c9041b4941c42dcc67c25260ccdbdaf3b0b47532bb07fc94b0a14\": container with ID starting with b269e69ef17c9041b4941c42dcc67c25260ccdbdaf3b0b47532bb07fc94b0a14 not found: ID does not exist" containerID="b269e69ef17c9041b4941c42dcc67c25260ccdbdaf3b0b47532bb07fc94b0a14"
Sep 08 13:40:19.974256 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.974191    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b269e69ef17c9041b4941c42dcc67c25260ccdbdaf3b0b47532bb07fc94b0a14"} err="failed to get container status \"b269e69ef17c9041b4941c42dcc67c25260ccdbdaf3b0b47532bb07fc94b0a14\": rpc error: code = NotFound desc = could not find container \"b269e69ef17c9041b4941c42dcc67c25260ccdbdaf3b0b47532bb07fc94b0a14\": container with ID starting with b269e69ef17c9041b4941c42dcc67c25260ccdbdaf3b0b47532bb07fc94b0a14 not found: ID does not exist"
Sep 08 13:40:19.974256 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.974215    2578 scope.go:117] "RemoveContainer" containerID="b9a668abed899b97c7b50244d789ff02f61b3a1f67629647128467a56b76503e"
Sep 08 13:40:19.974524 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:40:19.974497    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a668abed899b97c7b50244d789ff02f61b3a1f67629647128467a56b76503e\": container with ID starting with b9a668abed899b97c7b50244d789ff02f61b3a1f67629647128467a56b76503e not found: ID does not exist" containerID="b9a668abed899b97c7b50244d789ff02f61b3a1f67629647128467a56b76503e"
Sep 08 13:40:19.974578 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.974529    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a668abed899b97c7b50244d789ff02f61b3a1f67629647128467a56b76503e"} err="failed to get container status \"b9a668abed899b97c7b50244d789ff02f61b3a1f67629647128467a56b76503e\": rpc error: code = NotFound desc = could not find container \"b9a668abed899b97c7b50244d789ff02f61b3a1f67629647128467a56b76503e\": container with ID starting with b9a668abed899b97c7b50244d789ff02f61b3a1f67629647128467a56b76503e not found: ID does not exist"
Sep 08 13:40:19.978830 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.978807    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"]
Sep 08 13:40:19.985083 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:19.985031    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-7659888cb8-ktcnp"]
Sep 08 13:40:20.431046 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:20.431007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3b8372b-5073-41cc-8090-b7a72ad77146-proxy-tls\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:20.433486 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:20.433460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3b8372b-5073-41cc-8090-b7a72ad77146-proxy-tls\") pod \"isvc-canary-promote-predictor-64849fbd89-m42c6\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:20.649031 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:20.648997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:20.943770 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:20.943737    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"]
Sep 08 13:40:20.943882 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:40:20.943822    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b8372b_5073_41cc_8090_b7a72ad77146.slice/crio-cde725186e66a2b837cb97bac55109b51f67a8a709ee33e2a267625e100f9f3d WatchSource:0}: Error finding container cde725186e66a2b837cb97bac55109b51f67a8a709ee33e2a267625e100f9f3d: Status 404 returned error can't find the container with id cde725186e66a2b837cb97bac55109b51f67a8a709ee33e2a267625e100f9f3d
Sep 08 13:40:20.954645 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:20.954603    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" event={"ID":"b3b8372b-5073-41cc-8090-b7a72ad77146","Type":"ContainerStarted","Data":"cde725186e66a2b837cb97bac55109b51f67a8a709ee33e2a267625e100f9f3d"}
Sep 08 13:40:21.021569 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:21.021541    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2358e1b9-dd70-4b1f-a360-3da2cc3ded53" path="/var/lib/kubelet/pods/2358e1b9-dd70-4b1f-a360-3da2cc3ded53/volumes"
Sep 08 13:40:21.960648 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:21.960595    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" event={"ID":"b3b8372b-5073-41cc-8090-b7a72ad77146","Type":"ContainerStarted","Data":"e3320d8aaa1c486fb1f1173300132e97879d8fc71b7436e753cf5e0a95f98080"}
Sep 08 13:40:24.973882 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:24.973848    2578 generic.go:356] "Generic (PLEG): container finished" podID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerID="e3320d8aaa1c486fb1f1173300132e97879d8fc71b7436e753cf5e0a95f98080" exitCode=0
Sep 08 13:40:24.974263 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:24.973923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" event={"ID":"b3b8372b-5073-41cc-8090-b7a72ad77146","Type":"ContainerDied","Data":"e3320d8aaa1c486fb1f1173300132e97879d8fc71b7436e753cf5e0a95f98080"}
Sep 08 13:40:25.981335 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:25.981304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" event={"ID":"b3b8372b-5073-41cc-8090-b7a72ad77146","Type":"ContainerStarted","Data":"8ae98723c3d22bd2690c1283f435182d542eacd2c22fc1439f4d984ef89c51ce"}
Sep 08 13:40:25.981753 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:25.981367    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" event={"ID":"b3b8372b-5073-41cc-8090-b7a72ad77146","Type":"ContainerStarted","Data":"c9f38a63571f4d265295dd8e15e3026f044d9324eea7fbbf4f5a81c9f5de9fd7"}
Sep 08 13:40:25.981753 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:25.981640    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:26.005890 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:26.005849    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" podStartSLOduration=3.282874275 podStartE2EDuration="7.005834913s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.722960638s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:40:26.004738725 +0000 UTC m=+1443.604836371" watchObservedRunningTime="2026-09-08 13:40:26.005834913 +0000 UTC m=+1443.605932540"
Sep 08 13:40:26.986692 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:26.986658    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:26.987909 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:26.987874    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:40:27.990944 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:27.990900    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:40:32.995671 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:32.995640    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:40:32.996276 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:32.996247    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:40:42.996871 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:42.996817    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:40:52.996898 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:40:52.996849    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:41:02.996377 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:02.996338    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:41:12.996420 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:12.996381    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:41:22.997100 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:22.997067    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:41:32.997048 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:32.997003    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:41:34.008543 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:34.008483    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 08 13:41:42.061031 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:42.060994    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"]
Sep 08 13:41:42.061496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:42.061206    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kserve-container" containerID="cri-o://c9f38a63571f4d265295dd8e15e3026f044d9324eea7fbbf4f5a81c9f5de9fd7" gracePeriod=30
Sep 08 13:41:42.061496 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:42.061345    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kube-rbac-proxy" containerID="cri-o://8ae98723c3d22bd2690c1283f435182d542eacd2c22fc1439f4d984ef89c51ce" gracePeriod=30
Sep 08 13:41:42.062209 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:42.062188    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:41:42.295710 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:42.295672    2578 generic.go:356] "Generic (PLEG): container finished" podID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerID="8ae98723c3d22bd2690c1283f435182d542eacd2c22fc1439f4d984ef89c51ce" exitCode=2
Sep 08 13:41:42.295871 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:42.295744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" event={"ID":"b3b8372b-5073-41cc-8090-b7a72ad77146","Type":"ContainerDied","Data":"8ae98723c3d22bd2690c1283f435182d542eacd2c22fc1439f4d984ef89c51ce"}
Sep 08 13:41:42.991249 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:42.991211    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 08 13:41:46.597596 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.597570    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:41:46.740265 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.740236    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3b8372b-5073-41cc-8090-b7a72ad77146-kserve-provision-location\") pod \"b3b8372b-5073-41cc-8090-b7a72ad77146\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") "
Sep 08 13:41:46.740417 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.740308    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3b8372b-5073-41cc-8090-b7a72ad77146-kube-rbac-proxy-sar-config\") pod \"b3b8372b-5073-41cc-8090-b7a72ad77146\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") "
Sep 08 13:41:46.740417 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.740347    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3b8372b-5073-41cc-8090-b7a72ad77146-proxy-tls\") pod \"b3b8372b-5073-41cc-8090-b7a72ad77146\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") "
Sep 08 13:41:46.740417 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.740366    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psbkf\" (UniqueName: \"kubernetes.io/projected/b3b8372b-5073-41cc-8090-b7a72ad77146-kube-api-access-psbkf\") pod \"b3b8372b-5073-41cc-8090-b7a72ad77146\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") "
Sep 08 13:41:46.740417 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.740389    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3b8372b-5073-41cc-8090-b7a72ad77146-cabundle-cert\") pod \"b3b8372b-5073-41cc-8090-b7a72ad77146\" (UID: \"b3b8372b-5073-41cc-8090-b7a72ad77146\") "
Sep 08 13:41:46.740651 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.740585    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3b8372b-5073-41cc-8090-b7a72ad77146-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3b8372b-5073-41cc-8090-b7a72ad77146" (UID: "b3b8372b-5073-41cc-8090-b7a72ad77146"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:41:46.740813 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.740789    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b8372b-5073-41cc-8090-b7a72ad77146-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b3b8372b-5073-41cc-8090-b7a72ad77146" (UID: "b3b8372b-5073-41cc-8090-b7a72ad77146"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:41:46.740880 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.740805    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b8372b-5073-41cc-8090-b7a72ad77146-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b3b8372b-5073-41cc-8090-b7a72ad77146" (UID: "b3b8372b-5073-41cc-8090-b7a72ad77146"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:41:46.742484 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.742462    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3b8372b-5073-41cc-8090-b7a72ad77146-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b3b8372b-5073-41cc-8090-b7a72ad77146" (UID: "b3b8372b-5073-41cc-8090-b7a72ad77146"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:41:46.742591 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.742506    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b8372b-5073-41cc-8090-b7a72ad77146-kube-api-access-psbkf" (OuterVolumeSpecName: "kube-api-access-psbkf") pod "b3b8372b-5073-41cc-8090-b7a72ad77146" (UID: "b3b8372b-5073-41cc-8090-b7a72ad77146"). InnerVolumeSpecName "kube-api-access-psbkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:41:46.841831 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.841797    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3b8372b-5073-41cc-8090-b7a72ad77146-cabundle-cert\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:41:46.841831 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.841826    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3b8372b-5073-41cc-8090-b7a72ad77146-kserve-provision-location\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:41:46.841831 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.841837    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3b8372b-5073-41cc-8090-b7a72ad77146-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:41:46.842040 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.841846    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3b8372b-5073-41cc-8090-b7a72ad77146-proxy-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:41:46.842040 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:46.841855    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psbkf\" (UniqueName: \"kubernetes.io/projected/b3b8372b-5073-41cc-8090-b7a72ad77146-kube-api-access-psbkf\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:41:47.318467 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:47.318372    2578 generic.go:356] "Generic (PLEG): container finished" podID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerID="c9f38a63571f4d265295dd8e15e3026f044d9324eea7fbbf4f5a81c9f5de9fd7" exitCode=0
Sep 08 13:41:47.318467 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:47.318435    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"
Sep 08 13:41:47.318748 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:47.318457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" event={"ID":"b3b8372b-5073-41cc-8090-b7a72ad77146","Type":"ContainerDied","Data":"c9f38a63571f4d265295dd8e15e3026f044d9324eea7fbbf4f5a81c9f5de9fd7"}
Sep 08 13:41:47.318748 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:47.318539    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6" event={"ID":"b3b8372b-5073-41cc-8090-b7a72ad77146","Type":"ContainerDied","Data":"cde725186e66a2b837cb97bac55109b51f67a8a709ee33e2a267625e100f9f3d"}
Sep 08 13:41:47.318748 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:47.318580    2578 scope.go:117] "RemoveContainer" containerID="8ae98723c3d22bd2690c1283f435182d542eacd2c22fc1439f4d984ef89c51ce"
Sep 08 13:41:47.326724 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:47.326703    2578 scope.go:117] "RemoveContainer" containerID="c9f38a63571f4d265295dd8e15e3026f044d9324eea7fbbf4f5a81c9f5de9fd7"
Sep 08 13:41:47.334038 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:47.334018    2578 scope.go:117] "RemoveContainer" containerID="e3320d8aaa1c486fb1f1173300132e97879d8fc71b7436e753cf5e0a95f98080"
Sep 08 13:41:47.341594 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:47.341574    2578 scope.go:117] "RemoveContainer" containerID="8ae98723c3d22bd2690c1283f435182d542eacd2c22fc1439f4d984ef89c51ce"
Sep 08 13:41:47.341898 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:41:47.341876    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae98723c3d22bd2690c1283f435182d542eacd2c22fc1439f4d984ef89c51ce\": container with ID starting with 8ae98723c3d22bd2690c1283f435182d542eacd2c22fc1439f4d984ef89c51ce not found: ID does not exist" containerID="8ae98723c3d22bd2690c1283f435182d542eacd2c22fc1439f4d984ef89c51ce"
Sep 08 13:41:47.341984 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:47.341906    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae98723c3d22bd2690c1283f435182d542eacd2c22fc1439f4d984ef89c51ce"} err="failed to get container status \"8ae98723c3d22bd2690c1283f435182d542eacd2c22fc1439f4d984ef89c51ce\": rpc error: code = NotFound desc = could not find container \"8ae98723c3d22bd2690c1283f435182d542eacd2c22fc1439f4d984ef89c51ce\": container with ID starting with 8ae98723c3d22bd2690c1283f435182d542eacd2c22fc1439f4d984ef89c51ce not found: ID does not exist"
Sep 08 13:41:47.341984 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:47.341924    2578 scope.go:117] "RemoveContainer" containerID="c9f38a63571f4d265295dd8e15e3026f044d9324eea7fbbf4f5a81c9f5de9fd7"
Sep 08 13:41:47.342203 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:47.342184    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"]
Sep 08 13:41:47.342254 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:41:47.342187    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f38a63571f4d265295dd8e15e3026f044d9324eea7fbbf4f5a81c9f5de9fd7\": container with ID starting with c9f38a63571f4d265295dd8e15e3026f044d9324eea7fbbf4f5a81c9f5de9fd7 not found: ID does not exist" containerID="c9f38a63571f4d265295dd8e15e3026f044d9324eea7fbbf4f5a81c9f5de9fd7"
Sep 08 13:41:47.342254 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:47.342234    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f38a63571f4d265295dd8e15e3026f044d9324eea7fbbf4f5a81c9f5de9fd7"} err="failed to get container status \"c9f38a63571f4d265295dd8e15e3026f044d9324eea7fbbf4f5a81c9f5de9fd7\": rpc error: code = NotFound desc = could not find container \"c9f38a63571f4d265295dd8e15e3026f044d9324eea7fbbf4f5a81c9f5de9fd7\": container with ID starting with c9f38a63571f4d265295dd8e15e3026f044d9324eea7fbbf4f5a81c9f5de9fd7 not found: ID does not exist"
Sep 08 13:41:47.342254 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:47.342249    2578 scope.go:117] "RemoveContainer" containerID="e3320d8aaa1c486fb1f1173300132e97879d8fc71b7436e753cf5e0a95f98080"
Sep 08 13:41:47.342613 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:41:47.342547    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3320d8aaa1c486fb1f1173300132e97879d8fc71b7436e753cf5e0a95f98080\": container with ID starting with e3320d8aaa1c486fb1f1173300132e97879d8fc71b7436e753cf5e0a95f98080 not found: ID does not exist" containerID="e3320d8aaa1c486fb1f1173300132e97879d8fc71b7436e753cf5e0a95f98080"
Sep 08 13:41:47.342613 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:47.342580    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3320d8aaa1c486fb1f1173300132e97879d8fc71b7436e753cf5e0a95f98080"} err="failed to get container status \"e3320d8aaa1c486fb1f1173300132e97879d8fc71b7436e753cf5e0a95f98080\": rpc error: code = NotFound desc = could not find container \"e3320d8aaa1c486fb1f1173300132e97879d8fc71b7436e753cf5e0a95f98080\": container with ID starting with e3320d8aaa1c486fb1f1173300132e97879d8fc71b7436e753cf5e0a95f98080 not found: ID does not exist"
Sep 08 13:41:47.345391 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:47.345370    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-m42c6"]
Sep 08 13:41:49.012552 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.012513    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" path="/var/lib/kubelet/pods/b3b8372b-5073-41cc-8090-b7a72ad77146/volumes"
Sep 08 13:41:49.816725 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.816691    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"]
Sep 08 13:41:49.817045 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.817031    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kube-rbac-proxy"
Sep 08 13:41:49.817093 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.817047    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kube-rbac-proxy"
Sep 08 13:41:49.817093 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.817061    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kserve-container"
Sep 08 13:41:49.817093 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.817066    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kserve-container"
Sep 08 13:41:49.817093 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.817081    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="storage-initializer"
Sep 08 13:41:49.817093 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.817088    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="storage-initializer"
Sep 08 13:41:49.817425 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.817133    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kserve-container"
Sep 08 13:41:49.817425 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.817144    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3b8372b-5073-41cc-8090-b7a72ad77146" containerName="kube-rbac-proxy"
Sep 08 13:41:49.821821 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.821804    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:49.824595 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.824572    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:41:49.824752 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.824731    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-48hwc\""
Sep 08 13:41:49.825832 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.825759    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 08 13:41:49.825832 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.825797    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:41:49.825998 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.825799    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 08 13:41:49.825998 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.825802    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:41:49.825998 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.825766    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:41:49.831300 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.831280    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"]
Sep 08 13:41:49.966145 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.966104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a600f90-c374-44d7-aad2-8f86eb200f4d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-67454bd898-hwhlp\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:49.966334 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.966154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbcns\" (UniqueName: \"kubernetes.io/projected/6a600f90-c374-44d7-aad2-8f86eb200f4d-kube-api-access-lbcns\") pod \"isvc-canary-promote-v2-predictor-67454bd898-hwhlp\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:49.966334 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.966250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a600f90-c374-44d7-aad2-8f86eb200f4d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-67454bd898-hwhlp\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:49.966334 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.966303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a600f90-c374-44d7-aad2-8f86eb200f4d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-67454bd898-hwhlp\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:49.966334 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:49.966329    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a600f90-c374-44d7-aad2-8f86eb200f4d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-67454bd898-hwhlp\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:50.066873 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:50.066793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a600f90-c374-44d7-aad2-8f86eb200f4d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-67454bd898-hwhlp\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:50.066873 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:50.066832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbcns\" (UniqueName: \"kubernetes.io/projected/6a600f90-c374-44d7-aad2-8f86eb200f4d-kube-api-access-lbcns\") pod \"isvc-canary-promote-v2-predictor-67454bd898-hwhlp\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:50.066873 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:50.066863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a600f90-c374-44d7-aad2-8f86eb200f4d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-67454bd898-hwhlp\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:50.067325 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:50.066900    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a600f90-c374-44d7-aad2-8f86eb200f4d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-67454bd898-hwhlp\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:50.067325 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:50.066923    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a600f90-c374-44d7-aad2-8f86eb200f4d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-67454bd898-hwhlp\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:50.067325 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:50.067301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a600f90-c374-44d7-aad2-8f86eb200f4d-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-67454bd898-hwhlp\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:50.067576 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:50.067555    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a600f90-c374-44d7-aad2-8f86eb200f4d-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-67454bd898-hwhlp\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:50.067651 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:50.067607    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a600f90-c374-44d7-aad2-8f86eb200f4d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-67454bd898-hwhlp\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:50.069448 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:50.069425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a600f90-c374-44d7-aad2-8f86eb200f4d-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-67454bd898-hwhlp\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:50.076105 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:50.076085    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbcns\" (UniqueName: \"kubernetes.io/projected/6a600f90-c374-44d7-aad2-8f86eb200f4d-kube-api-access-lbcns\") pod \"isvc-canary-promote-v2-predictor-67454bd898-hwhlp\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:50.133136 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:50.133098    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:50.410364 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:50.410335    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"]
Sep 08 13:41:50.413084 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:41:50.413055    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a600f90_c374_44d7_aad2_8f86eb200f4d.slice/crio-d250ad273957eb02724a433e69c65098f5daac58b88805e166411893ad94a7fa WatchSource:0}: Error finding container d250ad273957eb02724a433e69c65098f5daac58b88805e166411893ad94a7fa: Status 404 returned error can't find the container with id d250ad273957eb02724a433e69c65098f5daac58b88805e166411893ad94a7fa
Sep 08 13:41:51.342150 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:51.342114    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" event={"ID":"6a600f90-c374-44d7-aad2-8f86eb200f4d","Type":"ContainerStarted","Data":"bd1773a553624ca1dee191507cf21f96160da2b859c21f5077a045b9d703dd05"}
Sep 08 13:41:51.342638 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:51.342178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" event={"ID":"6a600f90-c374-44d7-aad2-8f86eb200f4d","Type":"ContainerStarted","Data":"d250ad273957eb02724a433e69c65098f5daac58b88805e166411893ad94a7fa"}
Sep 08 13:41:54.354438 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:54.354403    2578 generic.go:356] "Generic (PLEG): container finished" podID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerID="bd1773a553624ca1dee191507cf21f96160da2b859c21f5077a045b9d703dd05" exitCode=0
Sep 08 13:41:54.354829 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:54.354473    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" event={"ID":"6a600f90-c374-44d7-aad2-8f86eb200f4d","Type":"ContainerDied","Data":"bd1773a553624ca1dee191507cf21f96160da2b859c21f5077a045b9d703dd05"}
Sep 08 13:41:55.361296 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:55.361262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" event={"ID":"6a600f90-c374-44d7-aad2-8f86eb200f4d","Type":"ContainerStarted","Data":"ad2ea148458dac9bd595f0eaf6eb971a90199ac2b4427a7a13efb9cf387747b0"}
Sep 08 13:41:55.361696 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:55.361325    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" event={"ID":"6a600f90-c374-44d7-aad2-8f86eb200f4d","Type":"ContainerStarted","Data":"ecc2fb372d6ff550bd77c0b6aef6503f515c99c2f86ba25269adedb85f75a286"}
Sep 08 13:41:55.361696 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:55.361568    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:55.396291 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:55.396241    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" podStartSLOduration=2.630196454 podStartE2EDuration="6.396224803s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.766028349s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:41:55.394010857 +0000 UTC m=+1532.994108483" watchObservedRunningTime="2026-09-08 13:41:55.396224803 +0000 UTC m=+1532.996322441"
Sep 08 13:41:56.365692 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:56.365654    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:41:56.366916 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:56.366889    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 08 13:41:57.370106 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:41:57.370065    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 08 13:42:02.374702 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:42:02.374672    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:42:02.375211 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:42:02.375178    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 08 13:42:12.376018 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:42:12.375976    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 08 13:42:22.375913 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:42:22.375872    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 08 13:42:32.375994 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:42:32.375955    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 08 13:42:42.375635 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:42:42.375571    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 08 13:42:52.375242 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:42:52.375205    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 08 13:43:02.376229 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:02.376182    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 08 13:43:12.375782 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:12.375750    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:43:20.155558 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:20.155468    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"]
Sep 08 13:43:20.156020 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:20.155799    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kserve-container" containerID="cri-o://ecc2fb372d6ff550bd77c0b6aef6503f515c99c2f86ba25269adedb85f75a286" gracePeriod=30
Sep 08 13:43:20.156020 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:20.155864    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kube-rbac-proxy" containerID="cri-o://ad2ea148458dac9bd595f0eaf6eb971a90199ac2b4427a7a13efb9cf387747b0" gracePeriod=30
Sep 08 13:43:20.715402 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:20.715365    2578 generic.go:356] "Generic (PLEG): container finished" podID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerID="ad2ea148458dac9bd595f0eaf6eb971a90199ac2b4427a7a13efb9cf387747b0" exitCode=2
Sep 08 13:43:20.715583 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:20.715435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" event={"ID":"6a600f90-c374-44d7-aad2-8f86eb200f4d","Type":"ContainerDied","Data":"ad2ea148458dac9bd595f0eaf6eb971a90199ac2b4427a7a13efb9cf387747b0"}
Sep 08 13:43:22.371093 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:22.371046    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 08 13:43:22.375787 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:22.375752    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 08 13:43:24.897045 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:24.897018    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:43:24.971935 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:24.971866    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a600f90-c374-44d7-aad2-8f86eb200f4d-kube-rbac-proxy-sar-config\") pod \"6a600f90-c374-44d7-aad2-8f86eb200f4d\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") "
Sep 08 13:43:24.971935 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:24.971901    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a600f90-c374-44d7-aad2-8f86eb200f4d-proxy-tls\") pod \"6a600f90-c374-44d7-aad2-8f86eb200f4d\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") "
Sep 08 13:43:24.971935 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:24.971935    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a600f90-c374-44d7-aad2-8f86eb200f4d-kserve-provision-location\") pod \"6a600f90-c374-44d7-aad2-8f86eb200f4d\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") "
Sep 08 13:43:24.972166 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:24.971964    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a600f90-c374-44d7-aad2-8f86eb200f4d-cabundle-cert\") pod \"6a600f90-c374-44d7-aad2-8f86eb200f4d\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") "
Sep 08 13:43:24.972166 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:24.972002    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbcns\" (UniqueName: \"kubernetes.io/projected/6a600f90-c374-44d7-aad2-8f86eb200f4d-kube-api-access-lbcns\") pod \"6a600f90-c374-44d7-aad2-8f86eb200f4d\" (UID: \"6a600f90-c374-44d7-aad2-8f86eb200f4d\") "
Sep 08 13:43:24.972293 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:24.972266    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a600f90-c374-44d7-aad2-8f86eb200f4d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a600f90-c374-44d7-aad2-8f86eb200f4d" (UID: "6a600f90-c374-44d7-aad2-8f86eb200f4d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:43:24.972344 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:24.972279    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a600f90-c374-44d7-aad2-8f86eb200f4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a600f90-c374-44d7-aad2-8f86eb200f4d" (UID: "6a600f90-c374-44d7-aad2-8f86eb200f4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:43:24.972344 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:24.972322    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a600f90-c374-44d7-aad2-8f86eb200f4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a600f90-c374-44d7-aad2-8f86eb200f4d" (UID: "6a600f90-c374-44d7-aad2-8f86eb200f4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:43:24.974022 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:24.973992    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a600f90-c374-44d7-aad2-8f86eb200f4d-kube-api-access-lbcns" (OuterVolumeSpecName: "kube-api-access-lbcns") pod "6a600f90-c374-44d7-aad2-8f86eb200f4d" (UID: "6a600f90-c374-44d7-aad2-8f86eb200f4d"). InnerVolumeSpecName "kube-api-access-lbcns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:43:24.974022 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:24.974007    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a600f90-c374-44d7-aad2-8f86eb200f4d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a600f90-c374-44d7-aad2-8f86eb200f4d" (UID: "6a600f90-c374-44d7-aad2-8f86eb200f4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:43:25.072584 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.072554    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a600f90-c374-44d7-aad2-8f86eb200f4d-kserve-provision-location\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:43:25.072584 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.072581    2578 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a600f90-c374-44d7-aad2-8f86eb200f4d-cabundle-cert\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:43:25.072584 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.072591    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbcns\" (UniqueName: \"kubernetes.io/projected/6a600f90-c374-44d7-aad2-8f86eb200f4d-kube-api-access-lbcns\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:43:25.072792 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.072600    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a600f90-c374-44d7-aad2-8f86eb200f4d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:43:25.072792 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.072610    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a600f90-c374-44d7-aad2-8f86eb200f4d-proxy-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:43:25.741525 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.741491    2578 generic.go:356] "Generic (PLEG): container finished" podID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerID="ecc2fb372d6ff550bd77c0b6aef6503f515c99c2f86ba25269adedb85f75a286" exitCode=0
Sep 08 13:43:25.741712 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.741529    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" event={"ID":"6a600f90-c374-44d7-aad2-8f86eb200f4d","Type":"ContainerDied","Data":"ecc2fb372d6ff550bd77c0b6aef6503f515c99c2f86ba25269adedb85f75a286"}
Sep 08 13:43:25.741712 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.741545    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"
Sep 08 13:43:25.741712 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.741584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp" event={"ID":"6a600f90-c374-44d7-aad2-8f86eb200f4d","Type":"ContainerDied","Data":"d250ad273957eb02724a433e69c65098f5daac58b88805e166411893ad94a7fa"}
Sep 08 13:43:25.741712 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.741612    2578 scope.go:117] "RemoveContainer" containerID="ad2ea148458dac9bd595f0eaf6eb971a90199ac2b4427a7a13efb9cf387747b0"
Sep 08 13:43:25.749723 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.749689    2578 scope.go:117] "RemoveContainer" containerID="ecc2fb372d6ff550bd77c0b6aef6503f515c99c2f86ba25269adedb85f75a286"
Sep 08 13:43:25.756668 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.756645    2578 scope.go:117] "RemoveContainer" containerID="bd1773a553624ca1dee191507cf21f96160da2b859c21f5077a045b9d703dd05"
Sep 08 13:43:25.762889 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.762868    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"]
Sep 08 13:43:25.764296 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.764282    2578 scope.go:117] "RemoveContainer" containerID="ad2ea148458dac9bd595f0eaf6eb971a90199ac2b4427a7a13efb9cf387747b0"
Sep 08 13:43:25.764593 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:43:25.764555    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad2ea148458dac9bd595f0eaf6eb971a90199ac2b4427a7a13efb9cf387747b0\": container with ID starting with ad2ea148458dac9bd595f0eaf6eb971a90199ac2b4427a7a13efb9cf387747b0 not found: ID does not exist" containerID="ad2ea148458dac9bd595f0eaf6eb971a90199ac2b4427a7a13efb9cf387747b0"
Sep 08 13:43:25.764738 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.764591    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad2ea148458dac9bd595f0eaf6eb971a90199ac2b4427a7a13efb9cf387747b0"} err="failed to get container status \"ad2ea148458dac9bd595f0eaf6eb971a90199ac2b4427a7a13efb9cf387747b0\": rpc error: code = NotFound desc = could not find container \"ad2ea148458dac9bd595f0eaf6eb971a90199ac2b4427a7a13efb9cf387747b0\": container with ID starting with ad2ea148458dac9bd595f0eaf6eb971a90199ac2b4427a7a13efb9cf387747b0 not found: ID does not exist"
Sep 08 13:43:25.764738 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.764614    2578 scope.go:117] "RemoveContainer" containerID="ecc2fb372d6ff550bd77c0b6aef6503f515c99c2f86ba25269adedb85f75a286"
Sep 08 13:43:25.764993 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:43:25.764972    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc2fb372d6ff550bd77c0b6aef6503f515c99c2f86ba25269adedb85f75a286\": container with ID starting with ecc2fb372d6ff550bd77c0b6aef6503f515c99c2f86ba25269adedb85f75a286 not found: ID does not exist" containerID="ecc2fb372d6ff550bd77c0b6aef6503f515c99c2f86ba25269adedb85f75a286"
Sep 08 13:43:25.765099 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.765000    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc2fb372d6ff550bd77c0b6aef6503f515c99c2f86ba25269adedb85f75a286"} err="failed to get container status \"ecc2fb372d6ff550bd77c0b6aef6503f515c99c2f86ba25269adedb85f75a286\": rpc error: code = NotFound desc = could not find container \"ecc2fb372d6ff550bd77c0b6aef6503f515c99c2f86ba25269adedb85f75a286\": container with ID starting with ecc2fb372d6ff550bd77c0b6aef6503f515c99c2f86ba25269adedb85f75a286 not found: ID does not exist"
Sep 08 13:43:25.765099 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.765020    2578 scope.go:117] "RemoveContainer" containerID="bd1773a553624ca1dee191507cf21f96160da2b859c21f5077a045b9d703dd05"
Sep 08 13:43:25.765305 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:43:25.765279    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd1773a553624ca1dee191507cf21f96160da2b859c21f5077a045b9d703dd05\": container with ID starting with bd1773a553624ca1dee191507cf21f96160da2b859c21f5077a045b9d703dd05 not found: ID does not exist" containerID="bd1773a553624ca1dee191507cf21f96160da2b859c21f5077a045b9d703dd05"
Sep 08 13:43:25.765401 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.765312    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1773a553624ca1dee191507cf21f96160da2b859c21f5077a045b9d703dd05"} err="failed to get container status \"bd1773a553624ca1dee191507cf21f96160da2b859c21f5077a045b9d703dd05\": rpc error: code = NotFound desc = could not find container \"bd1773a553624ca1dee191507cf21f96160da2b859c21f5077a045b9d703dd05\": container with ID starting with bd1773a553624ca1dee191507cf21f96160da2b859c21f5077a045b9d703dd05 not found: ID does not exist"
Sep 08 13:43:25.767373 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:25.767353    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-hwhlp"]
Sep 08 13:43:27.013744 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:27.013709    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" path="/var/lib/kubelet/pods/6a600f90-c374-44d7-aad2-8f86eb200f4d/volumes"
Sep 08 13:43:34.290845 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.290806    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"]
Sep 08 13:43:34.291336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.291123    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kserve-container"
Sep 08 13:43:34.291336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.291141    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kserve-container"
Sep 08 13:43:34.291336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.291172    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="storage-initializer"
Sep 08 13:43:34.291336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.291181    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="storage-initializer"
Sep 08 13:43:34.291336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.291190    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kube-rbac-proxy"
Sep 08 13:43:34.291336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.291198    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kube-rbac-proxy"
Sep 08 13:43:34.291336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.291260    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kserve-container"
Sep 08 13:43:34.291336 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.291270    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a600f90-c374-44d7-aad2-8f86eb200f4d" containerName="kube-rbac-proxy"
Sep 08 13:43:34.295826 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.295808    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:43:34.301711 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.301685    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 08 13:43:34.301851 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.301710    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-48hwc\""
Sep 08 13:43:34.301851 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.301741    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:43:34.302638 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.302606    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 08 13:43:34.302638 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.302610    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:43:34.314853 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.314830    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"]
Sep 08 13:43:34.446437 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.446402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4af3c4c-fef9-445a-9191-2ef36e4a922c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hgzbm\" (UID: \"f4af3c4c-fef9-445a-9191-2ef36e4a922c\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:43:34.446614 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.446447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4af3c4c-fef9-445a-9191-2ef36e4a922c-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hgzbm\" (UID: \"f4af3c4c-fef9-445a-9191-2ef36e4a922c\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:43:34.446614 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.446475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnzbc\" (UniqueName: \"kubernetes.io/projected/f4af3c4c-fef9-445a-9191-2ef36e4a922c-kube-api-access-lnzbc\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hgzbm\" (UID: \"f4af3c4c-fef9-445a-9191-2ef36e4a922c\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:43:34.547037 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.546959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnzbc\" (UniqueName: \"kubernetes.io/projected/f4af3c4c-fef9-445a-9191-2ef36e4a922c-kube-api-access-lnzbc\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hgzbm\" (UID: \"f4af3c4c-fef9-445a-9191-2ef36e4a922c\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:43:34.547037 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.547024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4af3c4c-fef9-445a-9191-2ef36e4a922c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hgzbm\" (UID: \"f4af3c4c-fef9-445a-9191-2ef36e4a922c\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:43:34.547222 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.547057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4af3c4c-fef9-445a-9191-2ef36e4a922c-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hgzbm\" (UID: \"f4af3c4c-fef9-445a-9191-2ef36e4a922c\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:43:34.547660 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.547610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4af3c4c-fef9-445a-9191-2ef36e4a922c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hgzbm\" (UID: \"f4af3c4c-fef9-445a-9191-2ef36e4a922c\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:43:34.549485 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.549468    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4af3c4c-fef9-445a-9191-2ef36e4a922c-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hgzbm\" (UID: \"f4af3c4c-fef9-445a-9191-2ef36e4a922c\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:43:34.555586 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.555567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnzbc\" (UniqueName: \"kubernetes.io/projected/f4af3c4c-fef9-445a-9191-2ef36e4a922c-kube-api-access-lnzbc\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-hgzbm\" (UID: \"f4af3c4c-fef9-445a-9191-2ef36e4a922c\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:43:34.606427 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.606402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:43:34.883944 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:34.883866    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"]
Sep 08 13:43:34.886940 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:43:34.886915    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4af3c4c_fef9_445a_9191_2ef36e4a922c.slice/crio-415443893dc76ffa1655c11ec8a2ae1286a8b6204e1c6d4cc15ecedfb316d359 WatchSource:0}: Error finding container 415443893dc76ffa1655c11ec8a2ae1286a8b6204e1c6d4cc15ecedfb316d359: Status 404 returned error can't find the container with id 415443893dc76ffa1655c11ec8a2ae1286a8b6204e1c6d4cc15ecedfb316d359
Sep 08 13:43:35.783825 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:35.783795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm" event={"ID":"f4af3c4c-fef9-445a-9191-2ef36e4a922c","Type":"ContainerStarted","Data":"8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b"}
Sep 08 13:43:35.784230 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:35.783848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm" event={"ID":"f4af3c4c-fef9-445a-9191-2ef36e4a922c","Type":"ContainerStarted","Data":"c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c"}
Sep 08 13:43:35.784230 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:35.783870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm" event={"ID":"f4af3c4c-fef9-445a-9191-2ef36e4a922c","Type":"ContainerStarted","Data":"415443893dc76ffa1655c11ec8a2ae1286a8b6204e1c6d4cc15ecedfb316d359"}
Sep 08 13:43:35.784230 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:35.783892    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:43:35.812152 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:35.812109    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm" podStartSLOduration=1.8120952799999999 podStartE2EDuration="1.81209528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:43:35.810367664 +0000 UTC m=+1633.410465291" watchObservedRunningTime="2026-09-08 13:43:35.81209528 +0000 UTC m=+1633.412192906"
Sep 08 13:43:36.791185 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:36.791147    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:43:36.793164 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:36.793138    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:43:43.804000 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:43:43.803967    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:45:09.350785 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:09.350750    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-hgzbm_f4af3c4c-fef9-445a-9191-2ef36e4a922c/kserve-container/0.log"
Sep 08 13:45:09.578613 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:09.578577    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"]
Sep 08 13:45:09.578841 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:09.578818    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm" podUID="f4af3c4c-fef9-445a-9191-2ef36e4a922c" containerName="kserve-container" containerID="cri-o://c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c" gracePeriod=30
Sep 08 13:45:09.578933 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:09.578898    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm" podUID="f4af3c4c-fef9-445a-9191-2ef36e4a922c" containerName="kube-rbac-proxy" containerID="cri-o://8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b" gracePeriod=30
Sep 08 13:45:09.818861 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:09.818834    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:45:09.938084 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:09.938045    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnzbc\" (UniqueName: \"kubernetes.io/projected/f4af3c4c-fef9-445a-9191-2ef36e4a922c-kube-api-access-lnzbc\") pod \"f4af3c4c-fef9-445a-9191-2ef36e4a922c\" (UID: \"f4af3c4c-fef9-445a-9191-2ef36e4a922c\") "
Sep 08 13:45:09.938274 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:09.938106    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4af3c4c-fef9-445a-9191-2ef36e4a922c-proxy-tls\") pod \"f4af3c4c-fef9-445a-9191-2ef36e4a922c\" (UID: \"f4af3c4c-fef9-445a-9191-2ef36e4a922c\") "
Sep 08 13:45:09.938274 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:09.938204    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4af3c4c-fef9-445a-9191-2ef36e4a922c-kube-rbac-proxy-sar-config\") pod \"f4af3c4c-fef9-445a-9191-2ef36e4a922c\" (UID: \"f4af3c4c-fef9-445a-9191-2ef36e4a922c\") "
Sep 08 13:45:09.938542 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:09.938515    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4af3c4c-fef9-445a-9191-2ef36e4a922c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f4af3c4c-fef9-445a-9191-2ef36e4a922c" (UID: "f4af3c4c-fef9-445a-9191-2ef36e4a922c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:45:09.940270 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:09.940220    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4af3c4c-fef9-445a-9191-2ef36e4a922c-kube-api-access-lnzbc" (OuterVolumeSpecName: "kube-api-access-lnzbc") pod "f4af3c4c-fef9-445a-9191-2ef36e4a922c" (UID: "f4af3c4c-fef9-445a-9191-2ef36e4a922c"). InnerVolumeSpecName "kube-api-access-lnzbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:45:09.940270 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:09.940228    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4af3c4c-fef9-445a-9191-2ef36e4a922c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4af3c4c-fef9-445a-9191-2ef36e4a922c" (UID: "f4af3c4c-fef9-445a-9191-2ef36e4a922c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:45:10.039297 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.039267    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4af3c4c-fef9-445a-9191-2ef36e4a922c-proxy-tls\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:45:10.039297 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.039293    2578 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4af3c4c-fef9-445a-9191-2ef36e4a922c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:45:10.039462 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.039306    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnzbc\" (UniqueName: \"kubernetes.io/projected/f4af3c4c-fef9-445a-9191-2ef36e4a922c-kube-api-access-lnzbc\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:45:10.178962 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.178934    2578 generic.go:356] "Generic (PLEG): container finished" podID="f4af3c4c-fef9-445a-9191-2ef36e4a922c" containerID="8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b" exitCode=2
Sep 08 13:45:10.178962 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.178956    2578 generic.go:356] "Generic (PLEG): container finished" podID="f4af3c4c-fef9-445a-9191-2ef36e4a922c" containerID="c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c" exitCode=2
Sep 08 13:45:10.179147 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.178988    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"
Sep 08 13:45:10.179147 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.178987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm" event={"ID":"f4af3c4c-fef9-445a-9191-2ef36e4a922c","Type":"ContainerDied","Data":"8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b"}
Sep 08 13:45:10.179147 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.179114    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm" event={"ID":"f4af3c4c-fef9-445a-9191-2ef36e4a922c","Type":"ContainerDied","Data":"c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c"}
Sep 08 13:45:10.179147 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.179137    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm" event={"ID":"f4af3c4c-fef9-445a-9191-2ef36e4a922c","Type":"ContainerDied","Data":"415443893dc76ffa1655c11ec8a2ae1286a8b6204e1c6d4cc15ecedfb316d359"}
Sep 08 13:45:10.179286 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.179160    2578 scope.go:117] "RemoveContainer" containerID="8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b"
Sep 08 13:45:10.187215 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.187193    2578 scope.go:117] "RemoveContainer" containerID="c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c"
Sep 08 13:45:10.194865 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.194846    2578 scope.go:117] "RemoveContainer" containerID="8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b"
Sep 08 13:45:10.195121 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:45:10.195102    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b\": container with ID starting with 8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b not found: ID does not exist" containerID="8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b"
Sep 08 13:45:10.195162 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.195130    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b"} err="failed to get container status \"8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b\": rpc error: code = NotFound desc = could not find container \"8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b\": container with ID starting with 8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b not found: ID does not exist"
Sep 08 13:45:10.195162 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.195148    2578 scope.go:117] "RemoveContainer" containerID="c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c"
Sep 08 13:45:10.195381 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:45:10.195366    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c\": container with ID starting with c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c not found: ID does not exist" containerID="c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c"
Sep 08 13:45:10.195424 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.195387    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c"} err="failed to get container status \"c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c\": rpc error: code = NotFound desc = could not find container \"c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c\": container with ID starting with c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c not found: ID does not exist"
Sep 08 13:45:10.195424 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.195402    2578 scope.go:117] "RemoveContainer" containerID="8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b"
Sep 08 13:45:10.195611 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.195593    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b"} err="failed to get container status \"8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b\": rpc error: code = NotFound desc = could not find container \"8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b\": container with ID starting with 8ee42ffe5f3e968989d0e38acda2dbe6f927f279e736ada4de90b1c37d27a18b not found: ID does not exist"
Sep 08 13:45:10.195679 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.195612    2578 scope.go:117] "RemoveContainer" containerID="c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c"
Sep 08 13:45:10.195897 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.195881    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c"} err="failed to get container status \"c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c\": rpc error: code = NotFound desc = could not find container \"c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c\": container with ID starting with c6303317fa091cc7466ae3e7cb09f6f42955bc2346f0986eb9ed431f9ecf7c5c not found: ID does not exist"
Sep 08 13:45:10.201761 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.201737    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"]
Sep 08 13:45:10.203543 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.203519    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-hgzbm"]
Sep 08 13:45:10.380788 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.380752    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2n2hd/must-gather-drb2q"]
Sep 08 13:45:10.381173 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.381056    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4af3c4c-fef9-445a-9191-2ef36e4a922c" containerName="kserve-container"
Sep 08 13:45:10.381173 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.381068    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4af3c4c-fef9-445a-9191-2ef36e4a922c" containerName="kserve-container"
Sep 08 13:45:10.381173 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.381084    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4af3c4c-fef9-445a-9191-2ef36e4a922c" containerName="kube-rbac-proxy"
Sep 08 13:45:10.381173 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.381089    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4af3c4c-fef9-445a-9191-2ef36e4a922c" containerName="kube-rbac-proxy"
Sep 08 13:45:10.381173 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.381141    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4af3c4c-fef9-445a-9191-2ef36e4a922c" containerName="kserve-container"
Sep 08 13:45:10.381173 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.381147    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4af3c4c-fef9-445a-9191-2ef36e4a922c" containerName="kube-rbac-proxy"
Sep 08 13:45:10.385262 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.385239    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2n2hd/must-gather-drb2q"
Sep 08 13:45:10.388355 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.388331    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2n2hd\"/\"openshift-service-ca.crt\""
Sep 08 13:45:10.388494 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.388397    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2n2hd\"/\"default-dockercfg-jkgrb\""
Sep 08 13:45:10.388971 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.388956    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2n2hd\"/\"kube-root-ca.crt\""
Sep 08 13:45:10.395974 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.395953    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2n2hd/must-gather-drb2q"]
Sep 08 13:45:10.443411 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.443383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sv4x\" (UniqueName: \"kubernetes.io/projected/7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a-kube-api-access-4sv4x\") pod \"must-gather-drb2q\" (UID: \"7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a\") " pod="openshift-must-gather-2n2hd/must-gather-drb2q"
Sep 08 13:45:10.443544 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.443428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a-must-gather-output\") pod \"must-gather-drb2q\" (UID: \"7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a\") " pod="openshift-must-gather-2n2hd/must-gather-drb2q"
Sep 08 13:45:10.544428 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.544336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sv4x\" (UniqueName: \"kubernetes.io/projected/7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a-kube-api-access-4sv4x\") pod \"must-gather-drb2q\" (UID: \"7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a\") " pod="openshift-must-gather-2n2hd/must-gather-drb2q"
Sep 08 13:45:10.544428 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.544388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a-must-gather-output\") pod \"must-gather-drb2q\" (UID: \"7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a\") " pod="openshift-must-gather-2n2hd/must-gather-drb2q"
Sep 08 13:45:10.544794 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.544774    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a-must-gather-output\") pod \"must-gather-drb2q\" (UID: \"7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a\") " pod="openshift-must-gather-2n2hd/must-gather-drb2q"
Sep 08 13:45:10.553364 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.553337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sv4x\" (UniqueName: \"kubernetes.io/projected/7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a-kube-api-access-4sv4x\") pod \"must-gather-drb2q\" (UID: \"7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a\") " pod="openshift-must-gather-2n2hd/must-gather-drb2q"
Sep 08 13:45:10.710909 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.710880    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2n2hd/must-gather-drb2q"
Sep 08 13:45:10.979852 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.979667    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2n2hd/must-gather-drb2q"]
Sep 08 13:45:10.982181 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:45:10.982157    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b7aa9b5_9aa3_4c6a_8973_9e7408739e6a.slice/crio-183f774e15ede0904312f3419fee11f936b46ebfd87d7780382e1a82d70ad8f9 WatchSource:0}: Error finding container 183f774e15ede0904312f3419fee11f936b46ebfd87d7780382e1a82d70ad8f9: Status 404 returned error can't find the container with id 183f774e15ede0904312f3419fee11f936b46ebfd87d7780382e1a82d70ad8f9
Sep 08 13:45:10.984340 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:10.984321    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:45:11.013645 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:11.013606    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4af3c4c-fef9-445a-9191-2ef36e4a922c" path="/var/lib/kubelet/pods/f4af3c4c-fef9-445a-9191-2ef36e4a922c/volumes"
Sep 08 13:45:11.185658 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:11.185612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2n2hd/must-gather-drb2q" event={"ID":"7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a","Type":"ContainerStarted","Data":"183f774e15ede0904312f3419fee11f936b46ebfd87d7780382e1a82d70ad8f9"}
Sep 08 13:45:18.225591 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:18.225548    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2n2hd/must-gather-drb2q" event={"ID":"7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a","Type":"ContainerStarted","Data":"99dbbb159834078a4287f9619a3be50caa0813ad1e34338e54c1080ec989b377"}
Sep 08 13:45:18.225591 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:18.225599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2n2hd/must-gather-drb2q" event={"ID":"7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a","Type":"ContainerStarted","Data":"c933eed9ca314684861b2d2d711bd49ce6c5cda12dfcdf270e8360c64bb38582"}
Sep 08 13:45:18.243874 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:18.243816    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2n2hd/must-gather-drb2q" podStartSLOduration=1.9344335190000002 podStartE2EDuration="8.243797637s" totalImagesPullingTime="6.309364118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:18.243295001 +0000 UTC m=+1735.843392627" watchObservedRunningTime="2026-09-08 13:45:18.243797637 +0000 UTC m=+1735.843895263"
Sep 08 13:45:23.042835 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:23.042799    2578 scope.go:117] "RemoveContainer" containerID="f56f02b0c225586d53d763b600adef9191e9b85bc60c59580243d74becd561e3"
Sep 08 13:45:23.052700 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:23.052328    2578 scope.go:117] "RemoveContainer" containerID="e20c632c8186a4fc8afd7f5f8fe4bc7639398b94a803b2c47b4415e4e028a602"
Sep 08 13:45:23.062043 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:23.062023    2578 scope.go:117] "RemoveContainer" containerID="f8f7bcba9c0723634891305a1b53e58244d819131908baf1de328a9a0bb2f7ed"
Sep 08 13:45:39.327502 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:39.327472    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" containerID="c933eed9ca314684861b2d2d711bd49ce6c5cda12dfcdf270e8360c64bb38582" exitCode=0
Sep 08 13:45:39.327920 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:39.327532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2n2hd/must-gather-drb2q" event={"ID":"7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a","Type":"ContainerDied","Data":"c933eed9ca314684861b2d2d711bd49ce6c5cda12dfcdf270e8360c64bb38582"}
Sep 08 13:45:39.327920 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:39.327841    2578 scope.go:117] "RemoveContainer" containerID="c933eed9ca314684861b2d2d711bd49ce6c5cda12dfcdf270e8360c64bb38582"
Sep 08 13:45:40.315896 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:40.315862    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2n2hd_must-gather-drb2q_7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a/gather/0.log"
Sep 08 13:45:41.020733 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:41.020700    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r7425/must-gather-9qzbc"]
Sep 08 13:45:41.022809 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:41.022791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r7425/must-gather-9qzbc"
Sep 08 13:45:41.025691 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:41.025672    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r7425\"/\"kube-root-ca.crt\""
Sep 08 13:45:41.025900 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:41.025883    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r7425\"/\"openshift-service-ca.crt\""
Sep 08 13:45:41.026365 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:41.026344    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r7425\"/\"default-dockercfg-zfwhh\""
Sep 08 13:45:41.038901 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:41.038878    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r7425/must-gather-9qzbc"]
Sep 08 13:45:41.111397 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:41.111363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/39006453-bd23-4278-a1cf-fcd0e602e4d5-must-gather-output\") pod \"must-gather-9qzbc\" (UID: \"39006453-bd23-4278-a1cf-fcd0e602e4d5\") " pod="openshift-must-gather-r7425/must-gather-9qzbc"
Sep 08 13:45:41.111532 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:41.111410    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp7tm\" (UniqueName: \"kubernetes.io/projected/39006453-bd23-4278-a1cf-fcd0e602e4d5-kube-api-access-cp7tm\") pod \"must-gather-9qzbc\" (UID: \"39006453-bd23-4278-a1cf-fcd0e602e4d5\") " pod="openshift-must-gather-r7425/must-gather-9qzbc"
Sep 08 13:45:41.212316 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:41.212288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/39006453-bd23-4278-a1cf-fcd0e602e4d5-must-gather-output\") pod \"must-gather-9qzbc\" (UID: \"39006453-bd23-4278-a1cf-fcd0e602e4d5\") " pod="openshift-must-gather-r7425/must-gather-9qzbc"
Sep 08 13:45:41.212464 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:41.212343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp7tm\" (UniqueName: \"kubernetes.io/projected/39006453-bd23-4278-a1cf-fcd0e602e4d5-kube-api-access-cp7tm\") pod \"must-gather-9qzbc\" (UID: \"39006453-bd23-4278-a1cf-fcd0e602e4d5\") " pod="openshift-must-gather-r7425/must-gather-9qzbc"
Sep 08 13:45:41.212644 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:41.212606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/39006453-bd23-4278-a1cf-fcd0e602e4d5-must-gather-output\") pod \"must-gather-9qzbc\" (UID: \"39006453-bd23-4278-a1cf-fcd0e602e4d5\") " pod="openshift-must-gather-r7425/must-gather-9qzbc"
Sep 08 13:45:41.224899 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:41.224873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp7tm\" (UniqueName: \"kubernetes.io/projected/39006453-bd23-4278-a1cf-fcd0e602e4d5-kube-api-access-cp7tm\") pod \"must-gather-9qzbc\" (UID: \"39006453-bd23-4278-a1cf-fcd0e602e4d5\") " pod="openshift-must-gather-r7425/must-gather-9qzbc"
Sep 08 13:45:41.331527 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:41.331450    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r7425/must-gather-9qzbc"
Sep 08 13:45:41.602829 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:41.602629    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r7425/must-gather-9qzbc"]
Sep 08 13:45:41.605440 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:45:41.605416    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39006453_bd23_4278_a1cf_fcd0e602e4d5.slice/crio-4153d5f1043c9242de735c1f4a5df1e1e979651bca4e72d4a5319c969705e303 WatchSource:0}: Error finding container 4153d5f1043c9242de735c1f4a5df1e1e979651bca4e72d4a5319c969705e303: Status 404 returned error can't find the container with id 4153d5f1043c9242de735c1f4a5df1e1e979651bca4e72d4a5319c969705e303
Sep 08 13:45:42.341799 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:42.341767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r7425/must-gather-9qzbc" event={"ID":"39006453-bd23-4278-a1cf-fcd0e602e4d5","Type":"ContainerStarted","Data":"4153d5f1043c9242de735c1f4a5df1e1e979651bca4e72d4a5319c969705e303"}
Sep 08 13:45:43.348462 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:43.348422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r7425/must-gather-9qzbc" event={"ID":"39006453-bd23-4278-a1cf-fcd0e602e4d5","Type":"ContainerStarted","Data":"84af1302e74212e1d27a4b720ea0d056e7c588992b44784730fc5f07fa4c053d"}
Sep 08 13:45:43.348875 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:43.348472    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r7425/must-gather-9qzbc" event={"ID":"39006453-bd23-4278-a1cf-fcd0e602e4d5","Type":"ContainerStarted","Data":"24a23f169ad791e359bee59ad3c9b2d963a75e310594ab41a4679e6e8bacddc4"}
Sep 08 13:45:43.380530 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:43.380464    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r7425/must-gather-9qzbc" podStartSLOduration=2.103823528 podStartE2EDuration="3.380443182s" totalImagesPullingTime="1.276619654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:43.37648636 +0000 UTC m=+1760.976583987" watchObservedRunningTime="2026-09-08 13:45:43.380443182 +0000 UTC m=+1760.980540811"
Sep 08 13:45:44.377675 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:44.377636    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5gvh2_dc805ec5-4549-4f72-9f0a-eb4e03cf22bb/global-pull-secret-syncer/0.log"
Sep 08 13:45:44.531395 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:44.531363    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p6sbc_4e40e15f-a782-46d6-b275-7722e86f4652/konnectivity-agent/0.log"
Sep 08 13:45:44.587217 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:44.587187    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-224.ec2.internal_bed360bd3f4f5cc25abd3af9aed5d2dd/haproxy/0.log"
Sep 08 13:45:45.844524 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:45.844484    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-2n2hd/must-gather-drb2q"]
Sep 08 13:45:45.847111 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:45.847081    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-2n2hd/must-gather-drb2q"]
Sep 08 13:45:45.847294 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:45.847267    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-2n2hd/must-gather-drb2q" podUID="7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" containerName="copy" containerID="cri-o://99dbbb159834078a4287f9619a3be50caa0813ad1e34338e54c1080ec989b377" gracePeriod=2
Sep 08 13:45:45.851011 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:45:45.850975    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-drb2q\" is forbidden: User \"system:node:ip-10-0-134-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2n2hd\": no relationship found between node 'ip-10-0-134-224.ec2.internal' and this object" podUID="7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" pod="openshift-must-gather-2n2hd/must-gather-drb2q"
Sep 08 13:45:46.206122 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.206022    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2n2hd_must-gather-drb2q_7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a/copy/0.log"
Sep 08 13:45:46.208427 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.208161    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2n2hd/must-gather-drb2q"
Sep 08 13:45:46.211828 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:45:46.211791    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-drb2q\" is forbidden: User \"system:node:ip-10-0-134-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2n2hd\": no relationship found between node 'ip-10-0-134-224.ec2.internal' and this object" podUID="7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" pod="openshift-must-gather-2n2hd/must-gather-drb2q"
Sep 08 13:45:46.256637 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.256408    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a-must-gather-output\") pod \"7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a\" (UID: \"7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a\") "
Sep 08 13:45:46.256637 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.256489    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sv4x\" (UniqueName: \"kubernetes.io/projected/7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a-kube-api-access-4sv4x\") pod \"7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a\" (UID: \"7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a\") "
Sep 08 13:45:46.259834 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.259799    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a-kube-api-access-4sv4x" (OuterVolumeSpecName: "kube-api-access-4sv4x") pod "7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" (UID: "7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a"). InnerVolumeSpecName "kube-api-access-4sv4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:45:46.263636 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.260180    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" (UID: "7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:46.358107 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.358065    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sv4x\" (UniqueName: \"kubernetes.io/projected/7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a-kube-api-access-4sv4x\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:45:46.358107 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.358106    2578 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a-must-gather-output\") on node \"ip-10-0-134-224.ec2.internal\" DevicePath \"\""
Sep 08 13:45:46.366382 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.366348    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2n2hd_must-gather-drb2q_7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a/copy/0.log"
Sep 08 13:45:46.366871 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.366841    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" containerID="99dbbb159834078a4287f9619a3be50caa0813ad1e34338e54c1080ec989b377" exitCode=143
Sep 08 13:45:46.366970 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.366924    2578 scope.go:117] "RemoveContainer" containerID="99dbbb159834078a4287f9619a3be50caa0813ad1e34338e54c1080ec989b377"
Sep 08 13:45:46.367101 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.367085    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2n2hd/must-gather-drb2q"
Sep 08 13:45:46.380424 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:45:46.380059    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-drb2q\" is forbidden: User \"system:node:ip-10-0-134-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2n2hd\": no relationship found between node 'ip-10-0-134-224.ec2.internal' and this object" podUID="7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" pod="openshift-must-gather-2n2hd/must-gather-drb2q"
Sep 08 13:45:46.398777 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:45:46.387494    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-drb2q\" is forbidden: User \"system:node:ip-10-0-134-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2n2hd\": no relationship found between node 'ip-10-0-134-224.ec2.internal' and this object" podUID="7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" pod="openshift-must-gather-2n2hd/must-gather-drb2q"
Sep 08 13:45:46.403175 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.403145    2578 scope.go:117] "RemoveContainer" containerID="c933eed9ca314684861b2d2d711bd49ce6c5cda12dfcdf270e8360c64bb38582"
Sep 08 13:45:46.440870 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.440738    2578 scope.go:117] "RemoveContainer" containerID="99dbbb159834078a4287f9619a3be50caa0813ad1e34338e54c1080ec989b377"
Sep 08 13:45:46.441486 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:45:46.441350    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99dbbb159834078a4287f9619a3be50caa0813ad1e34338e54c1080ec989b377\": container with ID starting with 99dbbb159834078a4287f9619a3be50caa0813ad1e34338e54c1080ec989b377 not found: ID does not exist" containerID="99dbbb159834078a4287f9619a3be50caa0813ad1e34338e54c1080ec989b377"
Sep 08 13:45:46.441486 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.441386    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99dbbb159834078a4287f9619a3be50caa0813ad1e34338e54c1080ec989b377"} err="failed to get container status \"99dbbb159834078a4287f9619a3be50caa0813ad1e34338e54c1080ec989b377\": rpc error: code = NotFound desc = could not find container \"99dbbb159834078a4287f9619a3be50caa0813ad1e34338e54c1080ec989b377\": container with ID starting with 99dbbb159834078a4287f9619a3be50caa0813ad1e34338e54c1080ec989b377 not found: ID does not exist"
Sep 08 13:45:46.441486 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.441411    2578 scope.go:117] "RemoveContainer" containerID="c933eed9ca314684861b2d2d711bd49ce6c5cda12dfcdf270e8360c64bb38582"
Sep 08 13:45:46.442043 ip-10-0-134-224 kubenswrapper[2578]: E0908 13:45:46.441977    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c933eed9ca314684861b2d2d711bd49ce6c5cda12dfcdf270e8360c64bb38582\": container with ID starting with c933eed9ca314684861b2d2d711bd49ce6c5cda12dfcdf270e8360c64bb38582 not found: ID does not exist" containerID="c933eed9ca314684861b2d2d711bd49ce6c5cda12dfcdf270e8360c64bb38582"
Sep 08 13:45:46.442043 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:46.442004    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c933eed9ca314684861b2d2d711bd49ce6c5cda12dfcdf270e8360c64bb38582"} err="failed to get container status \"c933eed9ca314684861b2d2d711bd49ce6c5cda12dfcdf270e8360c64bb38582\": rpc error: code = NotFound desc = could not find container \"c933eed9ca314684861b2d2d711bd49ce6c5cda12dfcdf270e8360c64bb38582\": container with ID starting with c933eed9ca314684861b2d2d711bd49ce6c5cda12dfcdf270e8360c64bb38582 not found: ID does not exist"
Sep 08 13:45:47.017590 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:47.017149    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" path="/var/lib/kubelet/pods/7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a/volumes"
Sep 08 13:45:48.820610 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:48.820502    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3376cf7e-4975-4688-a4ed-b5ac4acdf9f8/alertmanager/0.log"
Sep 08 13:45:48.850527 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:48.850498    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3376cf7e-4975-4688-a4ed-b5ac4acdf9f8/config-reloader/0.log"
Sep 08 13:45:48.884090 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:48.884037    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3376cf7e-4975-4688-a4ed-b5ac4acdf9f8/kube-rbac-proxy-web/0.log"
Sep 08 13:45:48.909914 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:48.909817    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3376cf7e-4975-4688-a4ed-b5ac4acdf9f8/kube-rbac-proxy/0.log"
Sep 08 13:45:48.936129 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:48.936077    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3376cf7e-4975-4688-a4ed-b5ac4acdf9f8/kube-rbac-proxy-metric/0.log"
Sep 08 13:45:48.965254 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:48.965155    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3376cf7e-4975-4688-a4ed-b5ac4acdf9f8/prom-label-proxy/0.log"
Sep 08 13:45:48.994444 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:48.994412    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3376cf7e-4975-4688-a4ed-b5ac4acdf9f8/init-config-reloader/0.log"
Sep 08 13:45:49.074369 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.074284    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-jj9fs_93c2041a-32e9-43bb-9d15-cf2acbcbafe4/kube-state-metrics/0.log"
Sep 08 13:45:49.110487 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.110452    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-jj9fs_93c2041a-32e9-43bb-9d15-cf2acbcbafe4/kube-rbac-proxy-main/0.log"
Sep 08 13:45:49.140935 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.140905    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-jj9fs_93c2041a-32e9-43bb-9d15-cf2acbcbafe4/kube-rbac-proxy-self/0.log"
Sep 08 13:45:49.208612 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.208576    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-b2l8f_9be4931c-c08f-4a6c-b84d-dcd6a9df9407/monitoring-plugin/0.log"
Sep 08 13:45:49.242256 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.242193    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cv55_4c3f1f57-7fe6-45a6-bd2d-e691ed68e756/node-exporter/0.log"
Sep 08 13:45:49.279647 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.276995    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cv55_4c3f1f57-7fe6-45a6-bd2d-e691ed68e756/kube-rbac-proxy/0.log"
Sep 08 13:45:49.305518 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.305483    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cv55_4c3f1f57-7fe6-45a6-bd2d-e691ed68e756/init-textfile/0.log"
Sep 08 13:45:49.521822 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.521789    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-pg47z_4584fb0f-fab8-4aae-b69f-98333a0b10ef/kube-rbac-proxy-main/0.log"
Sep 08 13:45:49.550637 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.550600    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-pg47z_4584fb0f-fab8-4aae-b69f-98333a0b10ef/kube-rbac-proxy-self/0.log"
Sep 08 13:45:49.577581 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.577547    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-pg47z_4584fb0f-fab8-4aae-b69f-98333a0b10ef/openshift-state-metrics/0.log"
Sep 08 13:45:49.624742 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.624694    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_008bac7a-8bcb-4a02-ba76-f85249da20be/prometheus/0.log"
Sep 08 13:45:49.651106 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.651065    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_008bac7a-8bcb-4a02-ba76-f85249da20be/config-reloader/0.log"
Sep 08 13:45:49.682417 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.682382    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_008bac7a-8bcb-4a02-ba76-f85249da20be/thanos-sidecar/0.log"
Sep 08 13:45:49.711064 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.711033    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_008bac7a-8bcb-4a02-ba76-f85249da20be/kube-rbac-proxy-web/0.log"
Sep 08 13:45:49.737166 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.737136    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_008bac7a-8bcb-4a02-ba76-f85249da20be/kube-rbac-proxy/0.log"
Sep 08 13:45:49.770254 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.770224    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_008bac7a-8bcb-4a02-ba76-f85249da20be/kube-rbac-proxy-thanos/0.log"
Sep 08 13:45:49.797013 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.796931    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_008bac7a-8bcb-4a02-ba76-f85249da20be/init-config-reloader/0.log"
Sep 08 13:45:49.828763 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.828730    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-f7k6w_59279b50-1274-4310-8b9d-4bd70bf37504/prometheus-operator/0.log"
Sep 08 13:45:49.853128 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.853088    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-f7k6w_59279b50-1274-4310-8b9d-4bd70bf37504/kube-rbac-proxy/0.log"
Sep 08 13:45:49.885829 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:49.885798    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-xl6b6_4e026921-5fe3-4637-93db-76890e2b0a69/prometheus-operator-admission-webhook/0.log"
Sep 08 13:45:53.023591 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.023556    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"]
Sep 08 13:45:53.024183 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.024006    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" containerName="copy"
Sep 08 13:45:53.024183 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.024024    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" containerName="copy"
Sep 08 13:45:53.024183 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.024035    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" containerName="gather"
Sep 08 13:45:53.024183 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.024040    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" containerName="gather"
Sep 08 13:45:53.024183 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.024089    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" containerName="gather"
Sep 08 13:45:53.024183 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.024097    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b7aa9b5-9aa3-4c6a-8973-9e7408739e6a" containerName="copy"
Sep 08 13:45:53.031804 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.031775    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.043505 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.043473    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"]
Sep 08 13:45:53.125809 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.125773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d095198a-7c00-461b-bf12-3deab61baadb-proc\") pod \"perf-node-gather-daemonset-bp2w9\" (UID: \"d095198a-7c00-461b-bf12-3deab61baadb\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.125981 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.125818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d095198a-7c00-461b-bf12-3deab61baadb-lib-modules\") pod \"perf-node-gather-daemonset-bp2w9\" (UID: \"d095198a-7c00-461b-bf12-3deab61baadb\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.126039 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.125989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d095198a-7c00-461b-bf12-3deab61baadb-sys\") pod \"perf-node-gather-daemonset-bp2w9\" (UID: \"d095198a-7c00-461b-bf12-3deab61baadb\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.126039 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.126033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkwvp\" (UniqueName: \"kubernetes.io/projected/d095198a-7c00-461b-bf12-3deab61baadb-kube-api-access-hkwvp\") pod \"perf-node-gather-daemonset-bp2w9\" (UID: \"d095198a-7c00-461b-bf12-3deab61baadb\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.126144 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.126061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d095198a-7c00-461b-bf12-3deab61baadb-podres\") pod \"perf-node-gather-daemonset-bp2w9\" (UID: \"d095198a-7c00-461b-bf12-3deab61baadb\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.227213 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.227156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d095198a-7c00-461b-bf12-3deab61baadb-sys\") pod \"perf-node-gather-daemonset-bp2w9\" (UID: \"d095198a-7c00-461b-bf12-3deab61baadb\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.227408 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.227229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwvp\" (UniqueName: \"kubernetes.io/projected/d095198a-7c00-461b-bf12-3deab61baadb-kube-api-access-hkwvp\") pod \"perf-node-gather-daemonset-bp2w9\" (UID: \"d095198a-7c00-461b-bf12-3deab61baadb\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.227408 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.227275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d095198a-7c00-461b-bf12-3deab61baadb-podres\") pod \"perf-node-gather-daemonset-bp2w9\" (UID: \"d095198a-7c00-461b-bf12-3deab61baadb\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.227408 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.227348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d095198a-7c00-461b-bf12-3deab61baadb-proc\") pod \"perf-node-gather-daemonset-bp2w9\" (UID: \"d095198a-7c00-461b-bf12-3deab61baadb\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.227408 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.227375    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d095198a-7c00-461b-bf12-3deab61baadb-lib-modules\") pod \"perf-node-gather-daemonset-bp2w9\" (UID: \"d095198a-7c00-461b-bf12-3deab61baadb\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.227668 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.227587    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d095198a-7c00-461b-bf12-3deab61baadb-lib-modules\") pod \"perf-node-gather-daemonset-bp2w9\" (UID: \"d095198a-7c00-461b-bf12-3deab61baadb\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.227738 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.227673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d095198a-7c00-461b-bf12-3deab61baadb-sys\") pod \"perf-node-gather-daemonset-bp2w9\" (UID: \"d095198a-7c00-461b-bf12-3deab61baadb\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.228200 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.228169    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d095198a-7c00-461b-bf12-3deab61baadb-podres\") pod \"perf-node-gather-daemonset-bp2w9\" (UID: \"d095198a-7c00-461b-bf12-3deab61baadb\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.228454 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.228435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d095198a-7c00-461b-bf12-3deab61baadb-proc\") pod \"perf-node-gather-daemonset-bp2w9\" (UID: \"d095198a-7c00-461b-bf12-3deab61baadb\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.237822 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.237795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwvp\" (UniqueName: \"kubernetes.io/projected/d095198a-7c00-461b-bf12-3deab61baadb-kube-api-access-hkwvp\") pod \"perf-node-gather-daemonset-bp2w9\" (UID: \"d095198a-7c00-461b-bf12-3deab61baadb\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.347206 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.347110    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:53.647053 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:53.647024    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"]
Sep 08 13:45:53.651993 ip-10-0-134-224 kubenswrapper[2578]: W0908 13:45:53.651951    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd095198a_7c00_461b_bf12_3deab61baadb.slice/crio-8778995f5f8689d4ea82580bad3712d567611b990273da59229a2c66a70313d8 WatchSource:0}: Error finding container 8778995f5f8689d4ea82580bad3712d567611b990273da59229a2c66a70313d8: Status 404 returned error can't find the container with id 8778995f5f8689d4ea82580bad3712d567611b990273da59229a2c66a70313d8
Sep 08 13:45:54.401078 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:54.401018    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gq26f_175b57a5-3977-4683-addd-174bfbfbbd24/dns/0.log"
Sep 08 13:45:54.415928 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:54.415899    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9" event={"ID":"d095198a-7c00-461b-bf12-3deab61baadb","Type":"ContainerStarted","Data":"51dc1c20f3258b2cc8f8bba292f4fe106f9f79aae9099f42bb341d3707368f4e"}
Sep 08 13:45:54.416084 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:54.415950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9" event={"ID":"d095198a-7c00-461b-bf12-3deab61baadb","Type":"ContainerStarted","Data":"8778995f5f8689d4ea82580bad3712d567611b990273da59229a2c66a70313d8"}
Sep 08 13:45:54.416084 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:54.416009    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:45:54.427471 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:54.427441    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gq26f_175b57a5-3977-4683-addd-174bfbfbbd24/kube-rbac-proxy/0.log"
Sep 08 13:45:54.435287 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:54.435245    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9" podStartSLOduration=2.4352307189999998 podStartE2EDuration="2.435230719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:54.433949136 +0000 UTC m=+1772.034046762" watchObservedRunningTime="2026-09-08 13:45:54.435230719 +0000 UTC m=+1772.035328345"
Sep 08 13:45:54.609949 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:54.609913    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j9nqf_1ba25c2e-e201-4764-adb2-e8faa0761d29/dns-node-resolver/0.log"
Sep 08 13:45:55.142721 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:55.142668    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kp8g5_a0f546ae-e7b5-4459-a35a-c16a0e7807b9/node-ca/0.log"
Sep 08 13:45:56.015183 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:56.015150    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f8f764c4b-6nkbv_e9e8b582-cc4d-40fe-a2c3-62538b991ba2/router/0.log"
Sep 08 13:45:56.369844 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:56.369771    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qfhsh_f0d9879c-70c8-4323-b19e-2e3c1c034566/serve-healthcheck-canary/0.log"
Sep 08 13:45:56.812539 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:56.812505    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-8v9kw_586b9989-b44f-4333-a970-66f5dd90f05f/insights-operator/0.log"
Sep 08 13:45:56.813476 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:56.813446    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-8v9kw_586b9989-b44f-4333-a970-66f5dd90f05f/insights-operator/1.log"
Sep 08 13:45:56.838244 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:56.838201    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7kmm7_07454356-ccf6-4e3e-8b6b-78473645fe8c/kube-rbac-proxy/0.log"
Sep 08 13:45:56.865987 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:56.865962    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7kmm7_07454356-ccf6-4e3e-8b6b-78473645fe8c/exporter/0.log"
Sep 08 13:45:56.895275 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:56.895257    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7kmm7_07454356-ccf6-4e3e-8b6b-78473645fe8c/extractor/0.log"
Sep 08 13:45:59.789090 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:59.789061    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-mwmwq_a7691dde-c1f7-4a28-9410-4b61288dc670/s3-init/0.log"
Sep 08 13:45:59.808185 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:45:59.808155    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-mwmwq_a7691dde-c1f7-4a28-9410-4b61288dc670/download-hf-model/0.log"
Sep 08 13:46:00.431879 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:00.431848    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r7425/perf-node-gather-daemonset-bp2w9"
Sep 08 13:46:05.761103 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:05.761070    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2jfd4_d7a807ed-4b45-45f2-a302-123e9b392de3/kube-multus/0.log"
Sep 08 13:46:05.862153 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:05.862119    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-95zgl_13d85014-a3c5-4627-80f6-62de2e79b1cf/kube-multus-additional-cni-plugins/0.log"
Sep 08 13:46:05.906185 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:05.906134    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-95zgl_13d85014-a3c5-4627-80f6-62de2e79b1cf/egress-router-binary-copy/0.log"
Sep 08 13:46:05.939260 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:05.939215    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-95zgl_13d85014-a3c5-4627-80f6-62de2e79b1cf/cni-plugins/0.log"
Sep 08 13:46:05.983486 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:05.983461    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-95zgl_13d85014-a3c5-4627-80f6-62de2e79b1cf/bond-cni-plugin/0.log"
Sep 08 13:46:06.017526 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:06.017467    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-95zgl_13d85014-a3c5-4627-80f6-62de2e79b1cf/routeoverride-cni/0.log"
Sep 08 13:46:06.043117 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:06.043078    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-95zgl_13d85014-a3c5-4627-80f6-62de2e79b1cf/whereabouts-cni-bincopy/0.log"
Sep 08 13:46:06.067440 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:06.067417    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-95zgl_13d85014-a3c5-4627-80f6-62de2e79b1cf/whereabouts-cni/0.log"
Sep 08 13:46:06.660660 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:06.660613    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h2td8_7eeefbda-9e69-4275-a756-6effb7738d90/network-metrics-daemon/0.log"
Sep 08 13:46:06.688251 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:06.688222    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-h2td8_7eeefbda-9e69-4275-a756-6effb7738d90/kube-rbac-proxy/0.log"
Sep 08 13:46:07.985267 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:07.985239    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms6p6_d609c6d5-329d-4100-951b-96638309f6cc/ovn-controller/0.log"
Sep 08 13:46:08.004592 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:08.004570    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms6p6_d609c6d5-329d-4100-951b-96638309f6cc/ovn-acl-logging/0.log"
Sep 08 13:46:08.031260 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:08.031229    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms6p6_d609c6d5-329d-4100-951b-96638309f6cc/kube-rbac-proxy-node/0.log"
Sep 08 13:46:08.054547 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:08.054523    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms6p6_d609c6d5-329d-4100-951b-96638309f6cc/kube-rbac-proxy-ovn-metrics/0.log"
Sep 08 13:46:08.075255 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:08.075233    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms6p6_d609c6d5-329d-4100-951b-96638309f6cc/northd/0.log"
Sep 08 13:46:08.099095 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:08.099074    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms6p6_d609c6d5-329d-4100-951b-96638309f6cc/nbdb/0.log"
Sep 08 13:46:08.122130 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:08.122107    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms6p6_d609c6d5-329d-4100-951b-96638309f6cc/sbdb/0.log"
Sep 08 13:46:08.162885 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:08.162866    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ms6p6_d609c6d5-329d-4100-951b-96638309f6cc/ovnkube-controller/0.log"
Sep 08 13:46:09.273956 ip-10-0-134-224 kubenswrapper[2578]: I0908 13:46:09.273922    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-csgtz_3f818ca3-1324-41d6-9d78-46c09e9719a3/network-check-target-container/0.log"