Jul 24 21:15:21.132412 ip-10-0-136-232 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 21:15:21.510414 ip-10-0-136-232 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 21:15:21.510414 ip-10-0-136-232 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 21:15:21.510414 ip-10-0-136-232 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 21:15:21.510414 ip-10-0-136-232 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 21:15:21.510414 ip-10-0-136-232 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 21:15:21.511893 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.511808    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 21:15:21.516579 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516563    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:21.516579 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516579    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516585    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516589    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516592    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516596    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516599    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516602    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516604    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516607    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516609    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516612    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516615    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516617    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516620    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516622    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516624    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516627    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516629    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516632    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:21.516654 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516634    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516640    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516643    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516646    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516648    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516651    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516653    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516655    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516659    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516661    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516664    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516667    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516669    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516672    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516674    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516677    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516679    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516682    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516684    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:21.517151 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516687    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516689    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516691    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516695    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516697    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516700    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516702    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516705    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516707    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516709    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516712    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516714    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516717    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516720    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516722    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516726    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516729    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516732    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516734    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516737    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:21.517686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516739    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516742    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516744    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516746    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516749    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516752    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516754    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516756    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516759    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516761    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516764    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516766    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516769    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516771    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516775    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516778    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516781    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516783    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516785    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516788    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:21.518213 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516791    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516793    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516795    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516799    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516801    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516804    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516808    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516812    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516815    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516817    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516820    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516823    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516825    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516828    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516830    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516833    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516835    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516837    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516840    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:21.518737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.516842    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517556    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517563    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517566    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517569    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517572    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517575    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517578    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517581    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517584    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517587    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517590    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517592    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517595    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517597    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517602    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517606    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517609    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517612    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:21.519242 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517616    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517618    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517621    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517624    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517627    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517630    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517632    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517635    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517638    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517640    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517642    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517645    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517647    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517650    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517653    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517655    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517657    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517661    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517663    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:21.519746 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517666    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517669    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517672    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517675    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517679    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517682    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517685    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517687    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517690    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517693    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517695    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517698    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517701    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517703    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517706    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517709    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517711    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517713    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517716    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517718    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:21.520269 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517721    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517723    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517725    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517728    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517730    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517732    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517735    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517738    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517741    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517747    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517749    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517752    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517754    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517757    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517760    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517762    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517765    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517768    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517770    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517773    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:21.520798 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517775    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517778    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517780    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517783    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517785    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517788    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517790    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517793    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517795    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517798    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517800    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517803    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517805    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517809    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517811    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517814    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517816    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517819    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517821    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:21.521338 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517824    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517826    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.517830    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518204    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518209    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518212    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518215    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518218    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518221    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518224    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518227    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518230    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518232    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518235    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518237    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518240    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518242    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518244    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518247    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518249    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:21.521842 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518252    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518256    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518259    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518262    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518265    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518268    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518270    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518273    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518275    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518278    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518280    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518283    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518285    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518287    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518291    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518294    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518296    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518298    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518301    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518303    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:21.522400 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518306    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518309    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518311    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518314    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518316    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518319    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518321    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518324    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518327    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518330    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518332    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518334    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518337    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518339    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518342    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518344    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518346    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518349    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518351    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:21.522907 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518354    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518357    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518361    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518363    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518368    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518371    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518374    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518377    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518380    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518382    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518385    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518387    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518390    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518393    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518396    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518398    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518400    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518403    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518405    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:21.523457 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518408    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518410    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518412    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518415    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518417    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518419    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518422    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518424    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518427    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518429    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518431    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518434    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518436    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518439    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518441    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518444    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518446    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518448    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518451    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518453    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:21.523947 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518457    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518459    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518462    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.518464    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518540    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518550    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518560    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518566    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518573    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518579    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518586    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518590    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518594    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518597    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518600    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518603    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518606    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518609    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518612    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518615    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518617    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518620    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518625    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518627    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 21:15:21.524481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518631    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518634    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518637    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518640    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518643    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518646    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518649    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518652    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518657    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518660    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518663    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518666    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518670    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518673    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518676    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518679    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518682    2560 flags.go:64] FLAG: --enable-server="true"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518685    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518689    2560 flags.go:64] FLAG: --event-burst="100"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518692    2560 flags.go:64] FLAG: --event-qps="50"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518695    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518698    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518700    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518704    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 21:15:21.525218 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518707    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518710    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518712    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518715    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518718    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518721    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518724    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518726    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518729    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518731    2560 flags.go:64] FLAG: --feature-gates=""
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518735    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518738    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518741    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518744    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518747    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518750    2560 flags.go:64] FLAG: --help="false"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518754    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-136-232.ec2.internal"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518757    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518759    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518762    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518765    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518769    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518772    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 21:15:21.525840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518774    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518777    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518780    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518783    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518786    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518789    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518791    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518794    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518797    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518799    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518802    2560 flags.go:64] FLAG: --lock-file=""
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518804    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518807    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518810    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518815    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518818    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518821    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518823    2560 flags.go:64] FLAG: --logging-format="text"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518827    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518830    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518833    2560 flags.go:64] FLAG: --manifest-url=""
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518835    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518839    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518842    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518846    2560 flags.go:64] FLAG: --max-pods="110"
Jul 24 21:15:21.526472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518849    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518852    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518855    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518858    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518862    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518865    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518867    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518875    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518878    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518881    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518884    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518887    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518892    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518895    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518898    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518901    2560 flags.go:64] FLAG: --port="10250"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518904    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518906    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b1353882a5afd2b5"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518909    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518912    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518915    2560 flags.go:64] FLAG: --register-node="true"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518918    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518921    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518924    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 24 21:15:21.527121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518926    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518929    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518932    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518935    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518938    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518941    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518943    2560 flags.go:64] FLAG: --runonce="false"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518946    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518949    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518952    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518955    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518957    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518960    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518964    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518967    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518970    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518973    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518975    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518978    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518981    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518984    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518986    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518992    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518994    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.518997    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 21:15:21.527771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.519001    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.519003    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.519006    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.519009    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.519011    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.519014    2560 flags.go:64] FLAG: --v="2"
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.519018    2560 flags.go:64] FLAG: --version="false"
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.519022    2560 flags.go:64] FLAG: --vmodule=""
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.519025    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.519028    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519151    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519155    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519158    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519161    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519164    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519167    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519171    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519173    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519176    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519178    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519183    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519186    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:21.528445 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519188    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519191    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519193    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519196    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519198    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519201    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519204    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519206    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519209    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519211    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519214    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519216    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519219    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519222    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519224    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519227    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519229    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519232    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519234    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519237    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:21.529045 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519239    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519241    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519244    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519246    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519249    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519253    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519256    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519258    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519262    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519266    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519269    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519272    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519276    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519279    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519282    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519285    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519288    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519290    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519293    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:21.529657 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519296    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519298    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519300    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519303    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519305    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519308    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519310    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519313    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519315    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519318    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519320    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519323    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519325    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519327    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519330    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519332    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519334    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519337    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519341    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519343    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:21.530171 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519345    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519348    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519350    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519354    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519356    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519358    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519361    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519363    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519366    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519368    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519371    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519373    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519376    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519379    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519382    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519384    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519387    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519389    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519392    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519394    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:21.530688 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519397    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:21.531214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519399    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:21.531214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519401    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:21.531214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519404    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:21.531214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519406    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:21.531214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519410    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:21.531214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519413    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:21.531214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.519416    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:21.531214 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.519867    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 21:15:21.531214 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.526940    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 21:15:21.531214 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.526958    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 21:15:21.531214 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.526975    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 21:15:21.531214 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.526980    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 21:15:21.531214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527052    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:21.531214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527058    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527062    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527064    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527067    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527070    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527072    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527075    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527077    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527081    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527085    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527089    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527094    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527098    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527101    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527103    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527106    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527110    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527113    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527116    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527119    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:21.531590 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527122    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527138    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527141    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527144    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527146    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527149    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527152    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527154    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527168    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527173    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527177    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527181    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527185    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527187    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527189    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527192    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527194    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527197    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527199    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527202    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:21.532214 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527204    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527206    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527209    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527211    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527213    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527216    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527219    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527223    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527226    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527228    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527231    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527234    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527236    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527239    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527241    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527245    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527249    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527253    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527257    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:21.532757 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527261    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527264    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527273    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527276    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527279    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527282    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527284    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527286    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527289    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527291    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527293    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527296    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527299    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527303    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527306    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527308    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527311    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527313    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527316    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527318    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:21.533374 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527321    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527324    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527328    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527332    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527336    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527341    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527344    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527347    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527349    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527351    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527354    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527356    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527359    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527362    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527364    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527373    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527376    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527378    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:21.533888 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527381    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:21.534405 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.527386    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 21:15:21.534405 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527559    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 21:15:21.534405 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527567    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 21:15:21.534405 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527580    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 21:15:21.534405 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527585    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 21:15:21.534405 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527589    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 21:15:21.534405 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527591    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 21:15:21.534405 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527594    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 21:15:21.534405 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527596    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 21:15:21.534405 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527599    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 21:15:21.534405 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527601    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 21:15:21.534405 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527604    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 21:15:21.534405 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527607    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 21:15:21.534405 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527610    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 21:15:21.534405 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527612    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527615    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527617    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527620    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527623    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527626    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527628    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527631    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527633    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527636    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527638    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527640    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527643    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527646    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527649    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527663    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527668    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527672    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527674    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 21:15:21.534928 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527677    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527679    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527682    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527684    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527686    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527689    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527691    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527693    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527696    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527698    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527701    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527703    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527705    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527708    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527710    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527712    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527716    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527719    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527722    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 21:15:21.535772 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527724    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527726    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527729    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527732    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527736    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527740    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527745    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527749    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527752    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527754    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527764    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527767    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527769    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527773    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527776    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527779    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527782    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527785    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527787    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 21:15:21.536340 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527790    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527794    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527797    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527800    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527802    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527805    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527807    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527809    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527812    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527817    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527821    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527825    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527829    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527833    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527835    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527838    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527840    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527842    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527844    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 21:15:21.537060 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527847    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 21:15:21.537786 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527849    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 21:15:21.537786 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527852    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 21:15:21.537786 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527854    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 21:15:21.537786 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527856    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 21:15:21.537786 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527865    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 21:15:21.537786 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527868    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 21:15:21.537786 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527870    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 21:15:21.537786 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527872    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 21:15:21.537786 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:21.527875    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 21:15:21.537786 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.527880    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 21:15:21.537786 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.528018    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 21:15:21.537786 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.531709    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 21:15:21.537786 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.532919    2560 server.go:1019] "Starting client certificate rotation"
Jul 24 21:15:21.537786 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.533039    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 21:15:21.538191 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.533080    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 21:15:21.538191 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.535703    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 21:15:21.541047 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.541027    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 21:15:21.551160 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.551139    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 24 21:15:21.554489 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.554473    2560 log.go:25] "Validated CRI v1 image API"
Jul 24 21:15:21.555311 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.555288    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 21:15:21.560739 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.560723    2560 fs.go:135] Filesystem UUIDs: map[065467c1-5294-45ed-b950-9c6f3533c990:/dev/nvme0n1p3 4e2e6411-f342-49bd-a86b-3db48e0afd8c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 24 21:15:21.560804 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.560740    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 21:15:21.565331 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.565309    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 21:15:21.565530 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.565436    2560 manager.go:216] Machine: {Timestamp:2026-07-24 21:15:21.564345935 +0000 UTC m=+0.320691740 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098648 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23bbbad4f30a81837f6bb290ab91e8 SystemUUID:ec23bbba-d4f3-0a81-837f-6bb290ab91e8 BootID:d61e7a6d-e262-45eb-9bac-f73cd39aefe2 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:16:63:d3:d4:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:16:63:d3:d4:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:aa:c7:06:07:fd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 21:15:21.565530 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.565530    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 21:15:21.565630 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.565603    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 21:15:21.566555 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.566533    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 21:15:21.566671 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.566557    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-232.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 21:15:21.567641 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.567630    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 21:15:21.567679 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.567644    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 21:15:21.567679 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.567651    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 21:15:21.568400 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.568389    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 21:15:21.569076 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.569067    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 21:15:21.571716 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.571706    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 21:15:21.571835 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.571826    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 21:15:21.571868 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.571854    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 21:15:21.571905 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.571870    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 21:15:21.571905 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.571885    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 21:15:21.571905 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.571895    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 24 21:15:21.571905 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.571903    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 21:15:21.572277 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.572265    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 21:15:21.572327 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.572284    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 21:15:21.574425 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.574408    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 21:15:21.575969 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.575956    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 21:15:21.576010 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.575986    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 21:15:21.577238 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.577225    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 21:15:21.577276 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.577250    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 21:15:21.577276 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.577259    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 21:15:21.577276 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.577264    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 21:15:21.577276 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.577272    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 21:15:21.577276 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.577277    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 21:15:21.577416 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.577285    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 21:15:21.577416 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.577290    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 21:15:21.577416 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.577296    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 21:15:21.577416 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.577301    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 21:15:21.577416 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.577309    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 21:15:21.577416 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.577314    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 21:15:21.577416 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.577336    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 21:15:21.577416 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.577346    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 21:15:21.580599 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.580490    2560 server.go:1268] "Started kubelet"
Jul 24 21:15:21.580599 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.580563    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 21:15:21.580731 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.580571    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 21:15:21.580781 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.580752    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 21:15:21.581256 ip-10-0-136-232 systemd[1]: Started Kubernetes Kubelet.
Jul 24 21:15:21.583535 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.583492    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 21:15:21.586011 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.585988    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 21:15:21.586370 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.586347    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-232.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 21:15:21.586553 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.586525    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 21:15:21.586683 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.586660    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-232.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 21:15:21.586934 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.586913    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 21:15:21.591900 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.590488    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-232.ec2.internal.18c5564e4a8ce9a5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-232.ec2.internal,UID:ip-10-0-136-232.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-232.ec2.internal,},FirstTimestamp:2026-07-24 21:15:21.580456357 +0000 UTC m=+0.336802161,LastTimestamp:2026-07-24 21:15:21.580456357 +0000 UTC m=+0.336802161,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-232.ec2.internal,}"
Jul 24 21:15:21.592650 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.592634    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p2c9p"
Jul 24 21:15:21.593327 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.593312    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 21:15:21.593399 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.593325    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 21:15:21.593522 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.593508    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-232.ec2.internal\" not found"
Jul 24 21:15:21.594195 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.594176    2560 factory.go:55] Registering systemd factory
Jul 24 21:15:21.594195 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.594198    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 24 21:15:21.594450 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.594335    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 21:15:21.594450 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.594394    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 21:15:21.594450 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.594411    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 21:15:21.594450 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.594427    2560 factory.go:153] Registering CRI-O factory
Jul 24 21:15:21.594450 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.594440    2560 factory.go:223] Registration of the crio container factory successfully
Jul 24 21:15:21.594662 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.594493    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 21:15:21.594662 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.594518    2560 factory.go:103] Registering Raw factory
Jul 24 21:15:21.594662 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.594533    2560 manager.go:1196] Started watching for new ooms in manager
Jul 24 21:15:21.594662 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.594537    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 21:15:21.594662 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.594545    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 21:15:21.595076 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.595058    2560 manager.go:349] Starting recovery of all containers
Jul 24 21:15:21.598026 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.597998    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-232.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 21:15:21.598153 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.598111    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 21:15:21.600498 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.600476    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p2c9p"
Jul 24 21:15:21.605053 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.604906    2560 manager.go:354] Recovery completed
Jul 24 21:15:21.609856 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.609840    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:21.611800 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.611783    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:21.611901 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.611814    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:21.611901 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.611827    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:21.612350 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.612336    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 21:15:21.612350 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.612349    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 21:15:21.612471 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.612362    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 21:15:21.612471 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.612373    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:21.613099 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.613084    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:21.613216 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.613114    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:21.613216 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.613152    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:21.614079 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.614022    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-232.ec2.internal.18c5564e4c6b2bf3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-232.ec2.internal,UID:ip-10-0-136-232.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-232.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-232.ec2.internal,},FirstTimestamp:2026-07-24 21:15:21.611799539 +0000 UTC m=+0.368145343,LastTimestamp:2026-07-24 21:15:21.611799539 +0000 UTC m=+0.368145343,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-232.ec2.internal,}"
Jul 24 21:15:21.614702 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.614691    2560 policy_none.go:49] "None policy: Start"
Jul 24 21:15:21.614743 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.614707    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 21:15:21.614743 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.614716    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 21:15:21.616672 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.616660    2560 policy_none.go:47] "Start"
Jul 24 21:15:21.653657 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.653641    2560 manager.go:337] "Starting Device Plugin manager"
Jul 24 21:15:21.653739 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.653670    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 21:15:21.653739 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.653680    2560 server.go:87] "Starting device plugin registration server"
Jul 24 21:15:21.653920 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.653908    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 21:15:21.653968 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.653923    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 21:15:21.654039 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.654020    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 21:15:21.654156 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.654143    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 21:15:21.654212 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.654157    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 21:15:21.654604 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.654588    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 21:15:21.654667 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.654631    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-232.ec2.internal\" not found"
Jul 24 21:15:21.746945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.746913    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 21:15:21.748949 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.748213    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 21:15:21.748949 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.748250    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 21:15:21.748949 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.748269    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 21:15:21.748949 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.748298    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 21:15:21.750410 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.750393    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:21.754492 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.754475    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:21.755163 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.755150    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:21.755233 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.755175    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:21.755233 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.755186    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:21.755233 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.755204    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-232.ec2.internal"
Jul 24 21:15:21.763636 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.763601    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-232.ec2.internal"
Jul 24 21:15:21.763636 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.763622    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-232.ec2.internal\": node \"ip-10-0-136-232.ec2.internal\" not found"
Jul 24 21:15:21.786231 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.786210    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-232.ec2.internal\" not found"
Jul 24 21:15:21.848380 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.848341    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-232.ec2.internal"]
Jul 24 21:15:21.848457 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.848414    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:21.849780 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.849761    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:21.849870 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.849795    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:21.849870 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.849808    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:21.851953 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.851941    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:21.852090 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.852077    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal"
Jul 24 21:15:21.852146 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.852106    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:21.852612 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.852596    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:21.852685 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.852623    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:21.852685 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.852600    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:21.852685 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.852659    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:21.852685 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.852672    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:21.852685 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.852632    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:21.854734 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.854719    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-232.ec2.internal"
Jul 24 21:15:21.854798 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.854746    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 21:15:21.855437 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.855415    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 21:15:21.855517 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.855450    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 21:15:21.855517 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.855474    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeHasSufficientPID"
Jul 24 21:15:21.878654 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.878631    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-232.ec2.internal\" not found" node="ip-10-0-136-232.ec2.internal"
Jul 24 21:15:21.883052 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.883036    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-232.ec2.internal\" not found" node="ip-10-0-136-232.ec2.internal"
Jul 24 21:15:21.886306 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.886292    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-232.ec2.internal\" not found"
Jul 24 21:15:21.896563 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.896548    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18a1aa0451307ff347116a4f0a729f93-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal\" (UID: \"18a1aa0451307ff347116a4f0a729f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal"
Jul 24 21:15:21.896646 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.896569    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e035e960d86bd45119eab5cda9361ecd-config\") pod \"kube-apiserver-proxy-ip-10-0-136-232.ec2.internal\" (UID: \"e035e960d86bd45119eab5cda9361ecd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-232.ec2.internal"
Jul 24 21:15:21.896646 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.896584    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18a1aa0451307ff347116a4f0a729f93-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal\" (UID: \"18a1aa0451307ff347116a4f0a729f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal"
Jul 24 21:15:21.987361 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:21.987325    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-232.ec2.internal\" not found"
Jul 24 21:15:21.997680 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.997658    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18a1aa0451307ff347116a4f0a729f93-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal\" (UID: \"18a1aa0451307ff347116a4f0a729f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal"
Jul 24 21:15:21.997759 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.997684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18a1aa0451307ff347116a4f0a729f93-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal\" (UID: \"18a1aa0451307ff347116a4f0a729f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal"
Jul 24 21:15:21.997759 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.997701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e035e960d86bd45119eab5cda9361ecd-config\") pod \"kube-apiserver-proxy-ip-10-0-136-232.ec2.internal\" (UID: \"e035e960d86bd45119eab5cda9361ecd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-232.ec2.internal"
Jul 24 21:15:21.997759 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.997739    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e035e960d86bd45119eab5cda9361ecd-config\") pod \"kube-apiserver-proxy-ip-10-0-136-232.ec2.internal\" (UID: \"e035e960d86bd45119eab5cda9361ecd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-232.ec2.internal"
Jul 24 21:15:21.997875 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.997775    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18a1aa0451307ff347116a4f0a729f93-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal\" (UID: \"18a1aa0451307ff347116a4f0a729f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal"
Jul 24 21:15:21.997875 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:21.997782    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18a1aa0451307ff347116a4f0a729f93-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal\" (UID: \"18a1aa0451307ff347116a4f0a729f93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal"
Jul 24 21:15:22.088117 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:22.088036    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-232.ec2.internal\" not found"
Jul 24 21:15:22.181613 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:22.181587    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal"
Jul 24 21:15:22.186421 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:22.186208    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-232.ec2.internal"
Jul 24 21:15:22.188340 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:22.188322    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-232.ec2.internal\" not found"
Jul 24 21:15:22.289400 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:22.289364    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-232.ec2.internal\" not found"
Jul 24 21:15:22.389855 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:22.389823    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-232.ec2.internal\" not found"
Jul 24 21:15:22.490448 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:22.490418    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-232.ec2.internal\" not found"
Jul 24 21:15:22.532674 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:22.532653    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 21:15:22.533138 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:22.532804    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 21:15:22.590639 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:22.590611    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-232.ec2.internal\" not found"
Jul 24 21:15:22.597511 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:22.597492    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:22.606348 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:22.606319    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 21:10:22 +0000 UTC" deadline="2028-03-15 07:40:21.803390352 +0000 UTC"
Jul 24 21:15:22.606348 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:22.606348    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14386h24m59.197046022s"
Jul 24 21:15:22.695014 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:22.694951    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-232.ec2.internal"
Jul 24 21:15:22.705941 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:22.705915    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 21:15:22.707628 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:22.707614    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal"
Jul 24 21:15:22.721682 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:22.721666    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 21:15:22.756272 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:22.756241    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a1aa0451307ff347116a4f0a729f93.slice/crio-a05b827009f3ac7f4ea8589c2633e79e5e338d6b579fd0567fcc5ef5ba0ba091 WatchSource:0}: Error finding container a05b827009f3ac7f4ea8589c2633e79e5e338d6b579fd0567fcc5ef5ba0ba091: Status 404 returned error can't find the container with id a05b827009f3ac7f4ea8589c2633e79e5e338d6b579fd0567fcc5ef5ba0ba091
Jul 24 21:15:22.761534 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:22.761514    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 21:15:22.766596 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:22.766572    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode035e960d86bd45119eab5cda9361ecd.slice/crio-6ca6e4d1a1807cb3b5853584173b2191607b4021c71d4bdc97b3ff7aaf8eb9b8 WatchSource:0}: Error finding container 6ca6e4d1a1807cb3b5853584173b2191607b4021c71d4bdc97b3ff7aaf8eb9b8: Status 404 returned error can't find the container with id 6ca6e4d1a1807cb3b5853584173b2191607b4021c71d4bdc97b3ff7aaf8eb9b8
Jul 24 21:15:23.056598 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.056510    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:23.192268 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.192226    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:23.573027 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.572738    2560 apiserver.go:52] "Watching apiserver"
Jul 24 21:15:23.581209 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.581179    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 21:15:23.581626 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.581595    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh","openshift-dns/node-resolver-hgz77","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal","openshift-multus/multus-additional-cni-plugins-pr7gn","openshift-ovn-kubernetes/ovnkube-node-88v79","kube-system/kube-apiserver-proxy-ip-10-0-136-232.ec2.internal","openshift-cluster-node-tuning-operator/tuned-p2n8v","openshift-image-registry/node-ca-p5ggk","openshift-multus/multus-5xnz9","openshift-multus/network-metrics-daemon-ldqjs","openshift-network-diagnostics/network-check-target-cpmnw","openshift-network-operator/iptables-alerter-fchvz","kube-system/global-pull-secret-syncer-b868q","kube-system/konnectivity-agent-2qcfz"]
Jul 24 21:15:23.584645 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.584620    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.587184 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.587165    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 21:15:23.587306 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.587199    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 21:15:23.587376 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.587311    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rxsn4\""
Jul 24 21:15:23.588368 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.588231    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:23.588368 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:23.588325    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:23.590732 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.590544    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2qcfz"
Jul 24 21:15:23.592873 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.592854    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.592987 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.592971    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pb5mq\""
Jul 24 21:15:23.594133 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.594113    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 21:15:23.594255 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.594227    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 21:15:23.594341 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.594327    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 21:15:23.594846 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.594824    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 21:15:23.595379 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.595361    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.596574 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.596556    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 21:15:23.596668 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.596586    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 21:15:23.596732 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.596698    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-spm82\""
Jul 24 21:15:23.596805 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.596792    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 21:15:23.596965 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.596949    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 21:15:23.597698 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.597556    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 21:15:23.597698 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.597588    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g7v74\""
Jul 24 21:15:23.598174 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.597835    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 21:15:23.598174 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.598119    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 21:15:23.598343 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.598234    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 21:15:23.598400 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.598377    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p5ggk"
Jul 24 21:15:23.598646 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.598626    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 21:15:23.599249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.599228    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 21:15:23.600005 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.599989    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.601790 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.601765    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 21:15:23.601923 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.601900    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 21:15:23.601985 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.601972    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7nhnc\""
Jul 24 21:15:23.602328 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.602307    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 21:15:23.602735 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.602712    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 21:15:23.603099 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.603078    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nts2h\""
Jul 24 21:15:23.606507 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.606482    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-slash\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.606600 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.606527    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-run-netns\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.606600 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.606563    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-lib-modules\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.606600 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.606596    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-tuned\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.606740 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.606628    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkppm\" (UniqueName: \"kubernetes.io/projected/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-kube-api-access-mkppm\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.606740 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.606661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-dbus\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:23.606740 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.606687    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-sys\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.606740 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.606717    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-tmp\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.606886 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.606751    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e02f6b0b-0d40-4e45-b729-bd64abdb2d64-agent-certs\") pod \"konnectivity-agent-2qcfz\" (UID: \"e02f6b0b-0d40-4e45-b729-bd64abdb2d64\") " pod="kube-system/konnectivity-agent-2qcfz"
Jul 24 21:15:23.606886 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.606809    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.606886 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.606849    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.606886 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.606878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-run-ovn\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.607031 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.606910    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-cni-netd\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.607031 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.606942    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb4843e9-e208-4283-bfb9-a5b4799e269f-ovnkube-script-lib\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.607031 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.606982    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-host\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.607031 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-var-lib-openvswitch\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.607182 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607047    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb4843e9-e208-4283-bfb9-a5b4799e269f-env-overrides\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.607182 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb4843e9-e208-4283-bfb9-a5b4799e269f-ovn-node-metrics-cert\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.607182 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607144    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-kubernetes\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.607182 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607179    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-sysctl-conf\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.607402 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607284    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gccvp\" (UniqueName: \"kubernetes.io/projected/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-kube-api-access-gccvp\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.607402 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-kubelet\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.607502 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607406    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-etc-openvswitch\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.607536 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607498    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-cni-bin\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.607586 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607559    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-run\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607704    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-var-lib-kubelet\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607785    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607831    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-cni-binary-copy\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607866    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-run-systemd\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-run-openvswitch\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.607932    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-node-log\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608047    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb4843e9-e208-4283-bfb9-a5b4799e269f-ovnkube-config\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608114    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl5xg\" (UniqueName: \"kubernetes.io/projected/cb4843e9-e208-4283-bfb9-a5b4799e269f-kube-api-access-vl5xg\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608150    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-sysconfig\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608214    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-sysctl-d\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-systemd\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608272    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608296    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-log-socket\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608339    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-run-ovn-kubernetes\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608386    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608396    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-kubelet-config\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:23.608945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608431    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-cnibin\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.609824 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608462    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.609824 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.609824 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608530    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-modprobe-d\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.609824 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608559    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e02f6b0b-0d40-4e45-b729-bd64abdb2d64-konnectivity-ca\") pod \"konnectivity-agent-2qcfz\" (UID: \"e02f6b0b-0d40-4e45-b729-bd64abdb2d64\") " pod="kube-system/konnectivity-agent-2qcfz"
Jul 24 21:15:23.609824 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608594    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-system-cni-dir\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.609824 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-os-release\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.609824 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.608791    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-systemd-units\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.610847 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.610723    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6s9bf\""
Jul 24 21:15:23.610847 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.610732    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 21:15:23.610847 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.610747    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 21:15:23.611053 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.610973    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 21:15:23.611053 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.610976    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hgz77"
Jul 24 21:15:23.613067 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.613048    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 21:15:23.613408 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.613388    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rh97l\""
Jul 24 21:15:23.613496 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.613453    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 21:15:23.613496 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.613391    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:23.613618 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:23.613540    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:23.615999 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.615982    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:23.616090 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:23.616041    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:23.618435 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.618410    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fchvz"
Jul 24 21:15:23.621084 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.620765    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 21:15:23.621084 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.620776    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 21:15:23.621084 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.620861    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 21:15:23.621084 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.620898    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r6gkj\""
Jul 24 21:15:23.628464 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.628443    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-222sl"
Jul 24 21:15:23.637014 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.636980    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-222sl"
Jul 24 21:15:23.696078 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.696054    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 21:15:23.709102 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709077    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-kubelet\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.709283 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709114    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-etc-openvswitch\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.709283 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709133    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-var-lib-kubelet\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.709283 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-cni-binary-copy\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.709283 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709242    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-etc-openvswitch\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.709283 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709269    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-var-lib-kubelet\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.709513 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709312    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-kubelet\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.709513 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709379    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1d3a096-c786-43b1-98fd-8e636a4c7e94-host-slash\") pod \"iptables-alerter-fchvz\" (UID: \"b1d3a096-c786-43b1-98fd-8e636a4c7e94\") " pod="openshift-network-operator/iptables-alerter-fchvz"
Jul 24 21:15:23.709513 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709414    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-run-openvswitch\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.709513 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709443    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl5xg\" (UniqueName: \"kubernetes.io/projected/cb4843e9-e208-4283-bfb9-a5b4799e269f-kube-api-access-vl5xg\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.709513 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709475    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-sysconfig\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.709513 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-run-openvswitch\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.709513 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709503    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-systemd\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.709817 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709555    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-sysconfig\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.709817 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709570    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-systemd\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.709817 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709591    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.709817 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709676    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-run-ovn-kubernetes\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.709817 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-cnibin\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.709817 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709732    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-multus-socket-dir-parent\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.709817 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709768    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-run-ovn-kubernetes\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.709817 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.709817 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e02f6b0b-0d40-4e45-b729-bd64abdb2d64-konnectivity-ca\") pod \"konnectivity-agent-2qcfz\" (UID: \"e02f6b0b-0d40-4e45-b729-bd64abdb2d64\") " pod="kube-system/konnectivity-agent-2qcfz"
Jul 24 21:15:23.710267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709768    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.710267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709845    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-system-cni-dir\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.710267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709859    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-cni-binary-copy\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.710267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709874    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-socket-dir\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.710267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709881    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-cnibin\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.710267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709905    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-systemd-units\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.710267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709932    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-dbus\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:23.710267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709943    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-system-cni-dir\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.710267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709957    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e02f6b0b-0d40-4e45-b729-bd64abdb2d64-agent-certs\") pod \"konnectivity-agent-2qcfz\" (UID: \"e02f6b0b-0d40-4e45-b729-bd64abdb2d64\") " pod="kube-system/konnectivity-agent-2qcfz"
Jul 24 21:15:23.710267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.709993    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.710267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-systemd-units\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.710267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-etc-kubernetes\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.710267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710122    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.710267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710231    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cfd5b405-1f77-48ea-a94f-c30b74583745-serviceca\") pod \"node-ca-p5ggk\" (UID: \"cfd5b405-1f77-48ea-a94f-c30b74583745\") " pod="openshift-image-registry/node-ca-p5ggk"
Jul 24 21:15:23.710267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710258    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-cnibin\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.710907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710290    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-run-k8s-cni-cncf-io\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.710907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-var-lib-kubelet\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.710907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710343    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd2kc\" (UniqueName: \"kubernetes.io/projected/62728d7d-6f8d-43dd-9824-67708026c9ff-kube-api-access-nd2kc\") pod \"node-resolver-hgz77\" (UID: \"62728d7d-6f8d-43dd-9824-67708026c9ff\") " pod="openshift-dns/node-resolver-hgz77"
Jul 24 21:15:23.710907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710371    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5pxp\" (UniqueName: \"kubernetes.io/projected/cfd5b405-1f77-48ea-a94f-c30b74583745-kube-api-access-s5pxp\") pod \"node-ca-p5ggk\" (UID: \"cfd5b405-1f77-48ea-a94f-c30b74583745\") " pod="openshift-image-registry/node-ca-p5ggk"
Jul 24 21:15:23.710907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjxg6\" (UniqueName: \"kubernetes.io/projected/b1d3a096-c786-43b1-98fd-8e636a4c7e94-kube-api-access-vjxg6\") pod \"iptables-alerter-fchvz\" (UID: \"b1d3a096-c786-43b1-98fd-8e636a4c7e94\") " pod="openshift-network-operator/iptables-alerter-fchvz"
Jul 24 21:15:23.710907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710413    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-dbus\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:23.710907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710425    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e02f6b0b-0d40-4e45-b729-bd64abdb2d64-konnectivity-ca\") pod \"konnectivity-agent-2qcfz\" (UID: \"e02f6b0b-0d40-4e45-b729-bd64abdb2d64\") " pod="kube-system/konnectivity-agent-2qcfz"
Jul 24 21:15:23.710907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-run-ovn\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.710907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710474    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-cni-netd\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.710907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710494    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-run-ovn\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.710907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710513    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb4843e9-e208-4283-bfb9-a5b4799e269f-ovnkube-script-lib\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.710907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710593    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-host\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.710907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.710907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710633    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-var-lib-cni-bin\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.710907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710648    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 21:15:23.711664 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.711664 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711061    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-cni-netd\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.711664 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710647    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-host\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.711664 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.710678    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-var-lib-cni-multus\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.711664 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711392    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-sys-fs\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.711664 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711425    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/62728d7d-6f8d-43dd-9824-67708026c9ff-hosts-file\") pod \"node-resolver-hgz77\" (UID: \"62728d7d-6f8d-43dd-9824-67708026c9ff\") " pod="openshift-dns/node-resolver-hgz77"
Jul 24 21:15:23.711664 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711455    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-var-lib-openvswitch\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.711664 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711488    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb4843e9-e208-4283-bfb9-a5b4799e269f-env-overrides\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.711664 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711520    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb4843e9-e208-4283-bfb9-a5b4799e269f-ovn-node-metrics-cert\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.711664 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711558    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb4843e9-e208-4283-bfb9-a5b4799e269f-ovnkube-script-lib\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.711664 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711610    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-kubernetes\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.711664 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-var-lib-openvswitch\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.712538 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711742    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-kubernetes\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.712538 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711787    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-sysctl-conf\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.712538 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711819    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccvp\" (UniqueName: \"kubernetes.io/projected/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-kube-api-access-gccvp\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.712538 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711856    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1aafe0ca-0b4a-400d-add6-3998833dd481-cni-binary-copy\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.712538 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-cni-bin\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.712538 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711933    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-run\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.712538 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711963    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:23.712538 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.711996    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-hostroot\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.712538 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.712034    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8fmd\" (UniqueName: \"kubernetes.io/projected/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-kube-api-access-m8fmd\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.712538 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.712070    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfd5b405-1f77-48ea-a94f-c30b74583745-host\") pod \"node-ca-p5ggk\" (UID: \"cfd5b405-1f77-48ea-a94f-c30b74583745\") " pod="openshift-image-registry/node-ca-p5ggk"
Jul 24 21:15:23.712538 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.712105    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mggb\" (UniqueName: \"kubernetes.io/projected/de44a92d-8440-4b1f-a0ba-479919425485-kube-api-access-8mggb\") pod \"network-metrics-daemon-ldqjs\" (UID: \"de44a92d-8440-4b1f-a0ba-479919425485\") " pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:23.712538 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.712207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-run-systemd\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.712538 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.712218    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-run\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.712538 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:23.712362    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.712557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb4843e9-e208-4283-bfb9-a5b4799e269f-env-overrides\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.712609    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-cni-bin\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.712625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-node-log\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.712672    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-sysctl-conf\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:23.712794    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret podName:1ec4ede0-f187-4d6c-a8a2-dd853f431b01 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:24.212759997 +0000 UTC m=+2.969105804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret") pod "global-pull-secret-syncer-b868q" (UID: "1ec4ede0-f187-4d6c-a8a2-dd853f431b01") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.712937    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb4843e9-e208-4283-bfb9-a5b4799e269f-ovnkube-config\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.712969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-sysctl-d\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.712972    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-node-log\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-system-cni-dir\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713071    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-run-multus-certs\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713101    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-sysctl-d\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713120    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-log-socket\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713181    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-run-systemd\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713206    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-log-socket\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713257    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-kubelet-config\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713293    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-os-release\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.713415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713326    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-device-dir\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713361    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1d3a096-c786-43b1-98fd-8e636a4c7e94-iptables-alerter-script\") pod \"iptables-alerter-fchvz\" (UID: \"b1d3a096-c786-43b1-98fd-8e636a4c7e94\") " pod="openshift-network-operator/iptables-alerter-fchvz"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713392    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-modprobe-d\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713406    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-kubelet-config\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713422    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-os-release\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713493    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-os-release\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs\") pod \"network-metrics-daemon-ldqjs\" (UID: \"de44a92d-8440-4b1f-a0ba-479919425485\") " pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-slash\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713612    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-run-netns\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713642    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-lib-modules\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713667    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb4843e9-e208-4283-bfb9-a5b4799e269f-ovnkube-config\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713671    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-tuned\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713722    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-modprobe-d\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713725    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkppm\" (UniqueName: \"kubernetes.io/projected/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-kube-api-access-mkppm\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713775    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713809    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-multus-cni-dir\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-run-netns\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.714432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.713988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-sys\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.715474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.714016    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-tmp\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.715474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.714108    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-multus-conf-dir\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.715474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.714186    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1aafe0ca-0b4a-400d-add6-3998833dd481-multus-daemon-config\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.715474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.714237    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-registration-dir\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.715474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.714296    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnslq\" (UniqueName: \"kubernetes.io/projected/1aafe0ca-0b4a-400d-add6-3998833dd481-kube-api-access-cnslq\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.715474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.714334    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9vcq\" (UniqueName: \"kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq\") pod \"network-check-target-cpmnw\" (UID: \"0f1880b1-1b8e-4633-841a-ed19c927a53d\") " pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:23.715474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.714358    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.715474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.714419    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.715474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.714482    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62728d7d-6f8d-43dd-9824-67708026c9ff-tmp-dir\") pod \"node-resolver-hgz77\" (UID: \"62728d7d-6f8d-43dd-9824-67708026c9ff\") " pod="openshift-dns/node-resolver-hgz77"
Jul 24 21:15:23.715474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.714605    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-slash\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.715474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.714704    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-lib-modules\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.715474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.714774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb4843e9-e208-4283-bfb9-a5b4799e269f-host-run-netns\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.715474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.714843    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-sys\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.716269 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.715814    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e02f6b0b-0d40-4e45-b729-bd64abdb2d64-agent-certs\") pod \"konnectivity-agent-2qcfz\" (UID: \"e02f6b0b-0d40-4e45-b729-bd64abdb2d64\") " pod="kube-system/konnectivity-agent-2qcfz"
Jul 24 21:15:23.716368 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.716345    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-etc-tuned\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.718159 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.718108    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-tmp\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.719045 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.718612    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb4843e9-e208-4283-bfb9-a5b4799e269f-ovn-node-metrics-cert\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.720823 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.720754    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl5xg\" (UniqueName: \"kubernetes.io/projected/cb4843e9-e208-4283-bfb9-a5b4799e269f-kube-api-access-vl5xg\") pod \"ovnkube-node-88v79\" (UID: \"cb4843e9-e208-4283-bfb9-a5b4799e269f\") " pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.721529 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.721508    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccvp\" (UniqueName: \"kubernetes.io/projected/a1be7ac6-f32e-40b4-8aa9-abca55380ae4-kube-api-access-gccvp\") pod \"multus-additional-cni-plugins-pr7gn\" (UID: \"a1be7ac6-f32e-40b4-8aa9-abca55380ae4\") " pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.725947 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.725928    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkppm\" (UniqueName: \"kubernetes.io/projected/dd8c44fe-e0bf-4ed2-8800-88ccba05358c-kube-api-access-mkppm\") pod \"tuned-p2n8v\" (UID: \"dd8c44fe-e0bf-4ed2-8800-88ccba05358c\") " pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.753420 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.753375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal" event={"ID":"18a1aa0451307ff347116a4f0a729f93","Type":"ContainerStarted","Data":"a05b827009f3ac7f4ea8589c2633e79e5e338d6b579fd0567fcc5ef5ba0ba091"}
Jul 24 21:15:23.754738 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.754714    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-232.ec2.internal" event={"ID":"e035e960d86bd45119eab5cda9361ecd","Type":"ContainerStarted","Data":"6ca6e4d1a1807cb3b5853584173b2191607b4021c71d4bdc97b3ff7aaf8eb9b8"}
Jul 24 21:15:23.815291 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815256    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-var-lib-cni-bin\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.815450 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815301    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-var-lib-cni-multus\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.815450 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815332    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-sys-fs\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.815450 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/62728d7d-6f8d-43dd-9824-67708026c9ff-hosts-file\") pod \"node-resolver-hgz77\" (UID: \"62728d7d-6f8d-43dd-9824-67708026c9ff\") " pod="openshift-dns/node-resolver-hgz77"
Jul 24 21:15:23.815450 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815375    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-var-lib-cni-multus\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.815450 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815392    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1aafe0ca-0b4a-400d-add6-3998833dd481-cni-binary-copy\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.815450 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815433    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-hostroot\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815449    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/62728d7d-6f8d-43dd-9824-67708026c9ff-hosts-file\") pod \"node-resolver-hgz77\" (UID: \"62728d7d-6f8d-43dd-9824-67708026c9ff\") " pod="openshift-dns/node-resolver-hgz77"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815453    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-sys-fs\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815376    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-var-lib-cni-bin\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815457    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8fmd\" (UniqueName: \"kubernetes.io/projected/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-kube-api-access-m8fmd\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfd5b405-1f77-48ea-a94f-c30b74583745-host\") pod \"node-ca-p5ggk\" (UID: \"cfd5b405-1f77-48ea-a94f-c30b74583745\") " pod="openshift-image-registry/node-ca-p5ggk"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mggb\" (UniqueName: \"kubernetes.io/projected/de44a92d-8440-4b1f-a0ba-479919425485-kube-api-access-8mggb\") pod \"network-metrics-daemon-ldqjs\" (UID: \"de44a92d-8440-4b1f-a0ba-479919425485\") " pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815547    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-hostroot\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-system-cni-dir\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815581    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-run-multus-certs\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815609    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-os-release\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815634    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-device-dir\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815667    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1d3a096-c786-43b1-98fd-8e636a4c7e94-iptables-alerter-script\") pod \"iptables-alerter-fchvz\" (UID: \"b1d3a096-c786-43b1-98fd-8e636a4c7e94\") " pod="openshift-network-operator/iptables-alerter-fchvz"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815695    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs\") pod \"network-metrics-daemon-ldqjs\" (UID: \"de44a92d-8440-4b1f-a0ba-479919425485\") " pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-multus-cni-dir\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815735    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfd5b405-1f77-48ea-a94f-c30b74583745-host\") pod \"node-ca-p5ggk\" (UID: \"cfd5b405-1f77-48ea-a94f-c30b74583745\") " pod="openshift-image-registry/node-ca-p5ggk"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-run-netns\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815773    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-multus-conf-dir\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.815841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815795    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-device-dir\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1aafe0ca-0b4a-400d-add6-3998833dd481-multus-daemon-config\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815841    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-registration-dir\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-system-cni-dir\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:23.815867    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815893    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-run-multus-certs\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:23.815939    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs podName:de44a92d-8440-4b1f-a0ba-479919425485 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:24.315919169 +0000 UTC m=+3.072264977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs") pod "network-metrics-daemon-ldqjs" (UID: "de44a92d-8440-4b1f-a0ba-479919425485") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815963    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-os-release\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815870    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnslq\" (UniqueName: \"kubernetes.io/projected/1aafe0ca-0b4a-400d-add6-3998833dd481-kube-api-access-cnslq\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.815982    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1aafe0ca-0b4a-400d-add6-3998833dd481-cni-binary-copy\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816005    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9vcq\" (UniqueName: \"kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq\") pod \"network-check-target-cpmnw\" (UID: \"0f1880b1-1b8e-4633-841a-ed19c927a53d\") " pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816060    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62728d7d-6f8d-43dd-9824-67708026c9ff-tmp-dir\") pod \"node-resolver-hgz77\" (UID: \"62728d7d-6f8d-43dd-9824-67708026c9ff\") " pod="openshift-dns/node-resolver-hgz77"
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816066    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-multus-conf-dir\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816104    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1d3a096-c786-43b1-98fd-8e636a4c7e94-host-slash\") pod \"iptables-alerter-fchvz\" (UID: \"b1d3a096-c786-43b1-98fd-8e636a4c7e94\") " pod="openshift-network-operator/iptables-alerter-fchvz"
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816143    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-multus-socket-dir-parent\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816175    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-socket-dir\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.816802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816238    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-etc-kubernetes\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816267    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816291    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cfd5b405-1f77-48ea-a94f-c30b74583745-serviceca\") pod \"node-ca-p5ggk\" (UID: \"cfd5b405-1f77-48ea-a94f-c30b74583745\") " pod="openshift-image-registry/node-ca-p5ggk"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816295    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-registration-dir\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816325    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-cnibin\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-run-k8s-cni-cncf-io\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816380    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-var-lib-kubelet\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816416    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd2kc\" (UniqueName: \"kubernetes.io/projected/62728d7d-6f8d-43dd-9824-67708026c9ff-kube-api-access-nd2kc\") pod \"node-resolver-hgz77\" (UID: \"62728d7d-6f8d-43dd-9824-67708026c9ff\") " pod="openshift-dns/node-resolver-hgz77"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816437    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816443    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5pxp\" (UniqueName: \"kubernetes.io/projected/cfd5b405-1f77-48ea-a94f-c30b74583745-kube-api-access-s5pxp\") pod \"node-ca-p5ggk\" (UID: \"cfd5b405-1f77-48ea-a94f-c30b74583745\") " pod="openshift-image-registry/node-ca-p5ggk"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816461    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-etc-kubernetes\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816490    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjxg6\" (UniqueName: \"kubernetes.io/projected/b1d3a096-c786-43b1-98fd-8e636a4c7e94-kube-api-access-vjxg6\") pod \"iptables-alerter-fchvz\" (UID: \"b1d3a096-c786-43b1-98fd-8e636a4c7e94\") " pod="openshift-network-operator/iptables-alerter-fchvz"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816530    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-multus-cni-dir\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816585    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-run-netns\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1d3a096-c786-43b1-98fd-8e636a4c7e94-iptables-alerter-script\") pod \"iptables-alerter-fchvz\" (UID: \"b1d3a096-c786-43b1-98fd-8e636a4c7e94\") " pod="openshift-network-operator/iptables-alerter-fchvz"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1aafe0ca-0b4a-400d-add6-3998833dd481-multus-daemon-config\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1d3a096-c786-43b1-98fd-8e636a4c7e94-host-slash\") pod \"iptables-alerter-fchvz\" (UID: \"b1d3a096-c786-43b1-98fd-8e636a4c7e94\") " pod="openshift-network-operator/iptables-alerter-fchvz"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-cnibin\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.817597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816640    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-multus-socket-dir-parent\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.818384 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816682    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.818384 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816702    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-run-k8s-cni-cncf-io\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.818384 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816732    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-socket-dir\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.818384 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.816770    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1aafe0ca-0b4a-400d-add6-3998833dd481-host-var-lib-kubelet\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.818384 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.817069    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cfd5b405-1f77-48ea-a94f-c30b74583745-serviceca\") pod \"node-ca-p5ggk\" (UID: \"cfd5b405-1f77-48ea-a94f-c30b74583745\") " pod="openshift-image-registry/node-ca-p5ggk"
Jul 24 21:15:23.818384 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.817108    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62728d7d-6f8d-43dd-9824-67708026c9ff-tmp-dir\") pod \"node-resolver-hgz77\" (UID: \"62728d7d-6f8d-43dd-9824-67708026c9ff\") " pod="openshift-dns/node-resolver-hgz77"
Jul 24 21:15:23.825536 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:23.825474    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:23.825536 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:23.825500    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:23.825536 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:23.825514    2560 projected.go:196] Error preparing data for projected volume kube-api-access-v9vcq for pod openshift-network-diagnostics/network-check-target-cpmnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:23.825763 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:23.825582    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq podName:0f1880b1-1b8e-4633-841a-ed19c927a53d nodeName:}" failed. No retries permitted until 2026-07-24 21:15:24.325565786 +0000 UTC m=+3.081911595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v9vcq" (UniqueName: "kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq") pod "network-check-target-cpmnw" (UID: "0f1880b1-1b8e-4633-841a-ed19c927a53d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:23.826163 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.826109    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8fmd\" (UniqueName: \"kubernetes.io/projected/b3a4ebda-c913-4bd7-8029-5a5a0a0571c4-kube-api-access-m8fmd\") pod \"aws-ebs-csi-driver-node-npjbh\" (UID: \"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.826485 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.826451    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mggb\" (UniqueName: \"kubernetes.io/projected/de44a92d-8440-4b1f-a0ba-479919425485-kube-api-access-8mggb\") pod \"network-metrics-daemon-ldqjs\" (UID: \"de44a92d-8440-4b1f-a0ba-479919425485\") " pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:23.826565 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.826498    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnslq\" (UniqueName: \"kubernetes.io/projected/1aafe0ca-0b4a-400d-add6-3998833dd481-kube-api-access-cnslq\") pod \"multus-5xnz9\" (UID: \"1aafe0ca-0b4a-400d-add6-3998833dd481\") " pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.827700 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.827675    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjxg6\" (UniqueName: \"kubernetes.io/projected/b1d3a096-c786-43b1-98fd-8e636a4c7e94-kube-api-access-vjxg6\") pod \"iptables-alerter-fchvz\" (UID: \"b1d3a096-c786-43b1-98fd-8e636a4c7e94\") " pod="openshift-network-operator/iptables-alerter-fchvz"
Jul 24 21:15:23.827952 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.827935    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd2kc\" (UniqueName: \"kubernetes.io/projected/62728d7d-6f8d-43dd-9824-67708026c9ff-kube-api-access-nd2kc\") pod \"node-resolver-hgz77\" (UID: \"62728d7d-6f8d-43dd-9824-67708026c9ff\") " pod="openshift-dns/node-resolver-hgz77"
Jul 24 21:15:23.828800 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.828773    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5pxp\" (UniqueName: \"kubernetes.io/projected/cfd5b405-1f77-48ea-a94f-c30b74583745-kube-api-access-s5pxp\") pod \"node-ca-p5ggk\" (UID: \"cfd5b405-1f77-48ea-a94f-c30b74583745\") " pod="openshift-image-registry/node-ca-p5ggk"
Jul 24 21:15:23.903796 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.903759    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p2n8v"
Jul 24 21:15:23.920741 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.920718    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2qcfz"
Jul 24 21:15:23.930649 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.930626    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pr7gn"
Jul 24 21:15:23.939741 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.939720    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:23.948516 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.948491    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p5ggk"
Jul 24 21:15:23.957443 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.957419    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5xnz9"
Jul 24 21:15:23.965159 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.965136    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh"
Jul 24 21:15:23.973961 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.973941    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hgz77"
Jul 24 21:15:23.981758 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:23.981722    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fchvz"
Jul 24 21:15:24.136138 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:24.136079    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 21:15:24.219679 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:24.219646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:24.219851 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:24.219765    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:24.219851 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:24.219836    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret podName:1ec4ede0-f187-4d6c-a8a2-dd853f431b01 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:25.219818914 +0000 UTC m=+3.976164711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret") pod "global-pull-secret-syncer-b868q" (UID: "1ec4ede0-f187-4d6c-a8a2-dd853f431b01") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:24.320583 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:24.320548    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs\") pod \"network-metrics-daemon-ldqjs\" (UID: \"de44a92d-8440-4b1f-a0ba-479919425485\") " pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:24.320760 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:24.320738    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:24.320847 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:24.320825    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs podName:de44a92d-8440-4b1f-a0ba-479919425485 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:25.320803882 +0000 UTC m=+4.077149688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs") pod "network-metrics-daemon-ldqjs" (UID: "de44a92d-8440-4b1f-a0ba-479919425485") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:24.421552 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:24.421487    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9vcq\" (UniqueName: \"kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq\") pod \"network-check-target-cpmnw\" (UID: \"0f1880b1-1b8e-4633-841a-ed19c927a53d\") " pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:24.421674 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:24.421651    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:24.421674 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:24.421673    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:24.421744 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:24.421683    2560 projected.go:196] Error preparing data for projected volume kube-api-access-v9vcq for pod openshift-network-diagnostics/network-check-target-cpmnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:24.421744 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:24.421737    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq podName:0f1880b1-1b8e-4633-841a-ed19c927a53d nodeName:}" failed. No retries permitted until 2026-07-24 21:15:25.421721782 +0000 UTC m=+4.178067575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9vcq" (UniqueName: "kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq") pod "network-check-target-cpmnw" (UID: "0f1880b1-1b8e-4633-841a-ed19c927a53d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:24.640830 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:24.640790    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 21:10:24 +0000 UTC" deadline="2028-01-31 02:19:54.531474026 +0000 UTC"
Jul 24 21:15:24.640830 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:24.640824    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13325h4m29.890655268s"
Jul 24 21:15:24.749103 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:24.749025    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:24.749103 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:24.749041    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:24.752869 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:24.749271    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:24.752869 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:24.749179    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:24.773343 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:24.773245    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4843e9_e208_4283_bfb9_a5b4799e269f.slice/crio-8c0525d0c7e8f390020f0479af08ef73449bebb419d97c9fe921f5ae5842e115 WatchSource:0}: Error finding container 8c0525d0c7e8f390020f0479af08ef73449bebb419d97c9fe921f5ae5842e115: Status 404 returned error can't find the container with id 8c0525d0c7e8f390020f0479af08ef73449bebb419d97c9fe921f5ae5842e115
Jul 24 21:15:24.795335 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:24.795306    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8c44fe_e0bf_4ed2_8800_88ccba05358c.slice/crio-269cd69192debaf8aa72cccbc115ba6cb62d96f9a5a3232be49dd4e39a0555d6 WatchSource:0}: Error finding container 269cd69192debaf8aa72cccbc115ba6cb62d96f9a5a3232be49dd4e39a0555d6: Status 404 returned error can't find the container with id 269cd69192debaf8aa72cccbc115ba6cb62d96f9a5a3232be49dd4e39a0555d6
Jul 24 21:15:24.796410 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:24.796387    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aafe0ca_0b4a_400d_add6_3998833dd481.slice/crio-6d3539c7edf1cb2851e166337edd85f2275ea1b038a8a56880bfa0881bcd8de7 WatchSource:0}: Error finding container 6d3539c7edf1cb2851e166337edd85f2275ea1b038a8a56880bfa0881bcd8de7: Status 404 returned error can't find the container with id 6d3539c7edf1cb2851e166337edd85f2275ea1b038a8a56880bfa0881bcd8de7
Jul 24 21:15:24.800686 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:24.800659    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd5b405_1f77_48ea_a94f_c30b74583745.slice/crio-1979fefb1dcb157d3d1187ec3e181d0def98c723f79c8ae9e1e79bbc34d7c8f6 WatchSource:0}: Error finding container 1979fefb1dcb157d3d1187ec3e181d0def98c723f79c8ae9e1e79bbc34d7c8f6: Status 404 returned error can't find the container with id 1979fefb1dcb157d3d1187ec3e181d0def98c723f79c8ae9e1e79bbc34d7c8f6
Jul 24 21:15:24.801009 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:24.800987    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62728d7d_6f8d_43dd_9824_67708026c9ff.slice/crio-486f24f44bd66c02598353ab3b72308bfc0c9f81d8f49e39016385f381703cb0 WatchSource:0}: Error finding container 486f24f44bd66c02598353ab3b72308bfc0c9f81d8f49e39016385f381703cb0: Status 404 returned error can't find the container with id 486f24f44bd66c02598353ab3b72308bfc0c9f81d8f49e39016385f381703cb0
Jul 24 21:15:24.806598 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:24.806571    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1be7ac6_f32e_40b4_8aa9_abca55380ae4.slice/crio-394721092aef12f0700b1afe41e86cd03d6ff503062eff187c59afd4206cc6c4 WatchSource:0}: Error finding container 394721092aef12f0700b1afe41e86cd03d6ff503062eff187c59afd4206cc6c4: Status 404 returned error can't find the container with id 394721092aef12f0700b1afe41e86cd03d6ff503062eff187c59afd4206cc6c4
Jul 24 21:15:24.807528 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:24.807504    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d3a096_c786_43b1_98fd_8e636a4c7e94.slice/crio-faf0c402fac668cf5afac48f167dd7084198328fd4ee7ff639c13c52227cfcf5 WatchSource:0}: Error finding container faf0c402fac668cf5afac48f167dd7084198328fd4ee7ff639c13c52227cfcf5: Status 404 returned error can't find the container with id faf0c402fac668cf5afac48f167dd7084198328fd4ee7ff639c13c52227cfcf5
Jul 24 21:15:24.809054 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:24.809025    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02f6b0b_0d40_4e45_b729_bd64abdb2d64.slice/crio-7475d86cbf509930f18c3b2bde4f66781030c56e9d8e5bb24935f3c72b6cdfd4 WatchSource:0}: Error finding container 7475d86cbf509930f18c3b2bde4f66781030c56e9d8e5bb24935f3c72b6cdfd4: Status 404 returned error can't find the container with id 7475d86cbf509930f18c3b2bde4f66781030c56e9d8e5bb24935f3c72b6cdfd4
Jul 24 21:15:25.230159 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.229412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:25.230159 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:25.229665    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:25.230159 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:25.229799    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret podName:1ec4ede0-f187-4d6c-a8a2-dd853f431b01 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:27.229775736 +0000 UTC m=+5.986121536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret") pod "global-pull-secret-syncer-b868q" (UID: "1ec4ede0-f187-4d6c-a8a2-dd853f431b01") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:25.330563 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.330473    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs\") pod \"network-metrics-daemon-ldqjs\" (UID: \"de44a92d-8440-4b1f-a0ba-479919425485\") " pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:25.330723 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:25.330690    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:25.330817 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:25.330756    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs podName:de44a92d-8440-4b1f-a0ba-479919425485 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:27.330736693 +0000 UTC m=+6.087082505 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs") pod "network-metrics-daemon-ldqjs" (UID: "de44a92d-8440-4b1f-a0ba-479919425485") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:25.431800 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.431756    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9vcq\" (UniqueName: \"kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq\") pod \"network-check-target-cpmnw\" (UID: \"0f1880b1-1b8e-4633-841a-ed19c927a53d\") " pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:25.431988 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:25.431962    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:25.431988 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:25.431983    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:25.432105 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:25.431997    2560 projected.go:196] Error preparing data for projected volume kube-api-access-v9vcq for pod openshift-network-diagnostics/network-check-target-cpmnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:25.432105 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:25.432059    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq podName:0f1880b1-1b8e-4633-841a-ed19c927a53d nodeName:}" failed. No retries permitted until 2026-07-24 21:15:27.432037609 +0000 UTC m=+6.188383408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9vcq" (UniqueName: "kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq") pod "network-check-target-cpmnw" (UID: "0f1880b1-1b8e-4633-841a-ed19c927a53d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:25.641360 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.641293    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 21:10:24 +0000 UTC" deadline="2028-04-03 21:52:09.287435404 +0000 UTC"
Jul 24 21:15:25.641360 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.641358    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14856h36m43.64608167s"
Jul 24 21:15:25.751802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.751409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:25.751802 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:25.751594    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:25.788753 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.788713    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-232.ec2.internal" event={"ID":"e035e960d86bd45119eab5cda9361ecd","Type":"ContainerStarted","Data":"d4404417c22e002c4f8cb3f72e2707f8f169c964cb508685f089e1a90c9a5c18"}
Jul 24 21:15:25.809303 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.809266    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2qcfz" event={"ID":"e02f6b0b-0d40-4e45-b729-bd64abdb2d64","Type":"ContainerStarted","Data":"7475d86cbf509930f18c3b2bde4f66781030c56e9d8e5bb24935f3c72b6cdfd4"}
Jul 24 21:15:25.821718 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.821627    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hgz77" event={"ID":"62728d7d-6f8d-43dd-9824-67708026c9ff","Type":"ContainerStarted","Data":"486f24f44bd66c02598353ab3b72308bfc0c9f81d8f49e39016385f381703cb0"}
Jul 24 21:15:25.831897 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.831691    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p5ggk" event={"ID":"cfd5b405-1f77-48ea-a94f-c30b74583745","Type":"ContainerStarted","Data":"1979fefb1dcb157d3d1187ec3e181d0def98c723f79c8ae9e1e79bbc34d7c8f6"}
Jul 24 21:15:25.837705 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.837674    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5xnz9" event={"ID":"1aafe0ca-0b4a-400d-add6-3998833dd481","Type":"ContainerStarted","Data":"6d3539c7edf1cb2851e166337edd85f2275ea1b038a8a56880bfa0881bcd8de7"}
Jul 24 21:15:25.850315 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.850285    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88v79" event={"ID":"cb4843e9-e208-4283-bfb9-a5b4799e269f","Type":"ContainerStarted","Data":"8c0525d0c7e8f390020f0479af08ef73449bebb419d97c9fe921f5ae5842e115"}
Jul 24 21:15:25.863227 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.863175    2560 generic.go:356] "Generic (PLEG): container finished" podID="18a1aa0451307ff347116a4f0a729f93" containerID="0a814716e081e4035c7e16a0c04fbbb2c66e80515712966549c0df2a545315b2" exitCode=0
Jul 24 21:15:25.864182 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.863955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal" event={"ID":"18a1aa0451307ff347116a4f0a729f93","Type":"ContainerDied","Data":"0a814716e081e4035c7e16a0c04fbbb2c66e80515712966549c0df2a545315b2"}
Jul 24 21:15:25.882483 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.882448    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fchvz" event={"ID":"b1d3a096-c786-43b1-98fd-8e636a4c7e94","Type":"ContainerStarted","Data":"faf0c402fac668cf5afac48f167dd7084198328fd4ee7ff639c13c52227cfcf5"}
Jul 24 21:15:25.883066 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.883014    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-232.ec2.internal" podStartSLOduration=2.882993732 podStartE2EDuration="2.882993732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:25.806969933 +0000 UTC m=+4.563315751" watchObservedRunningTime="2026-07-24 21:15:25.882993732 +0000 UTC m=+4.639339575"
Jul 24 21:15:25.895287 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.895151    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh" event={"ID":"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4","Type":"ContainerStarted","Data":"8293129ddc86559d301ef6f0b3a92a66118592d260e64461837274082cba40ec"}
Jul 24 21:15:25.899771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.899698    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pr7gn" event={"ID":"a1be7ac6-f32e-40b4-8aa9-abca55380ae4","Type":"ContainerStarted","Data":"394721092aef12f0700b1afe41e86cd03d6ff503062eff187c59afd4206cc6c4"}
Jul 24 21:15:25.904374 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:25.904294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p2n8v" event={"ID":"dd8c44fe-e0bf-4ed2-8800-88ccba05358c","Type":"ContainerStarted","Data":"269cd69192debaf8aa72cccbc115ba6cb62d96f9a5a3232be49dd4e39a0555d6"}
Jul 24 21:15:26.748755 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:26.748719    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:26.749333 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:26.748872    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:26.749629 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:26.749605    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:26.749768 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:26.749743    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:26.921707 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:26.921635    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal" event={"ID":"18a1aa0451307ff347116a4f0a729f93","Type":"ContainerStarted","Data":"fe0e0cb8da4fcb9359d3417e884ac110f5a23c549f1b68cda4f20181a332dd0d"}
Jul 24 21:15:27.246856 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:27.246820    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:27.247063 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:27.247011    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:27.247138 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:27.247081    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret podName:1ec4ede0-f187-4d6c-a8a2-dd853f431b01 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:31.247059514 +0000 UTC m=+10.003405307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret") pod "global-pull-secret-syncer-b868q" (UID: "1ec4ede0-f187-4d6c-a8a2-dd853f431b01") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:27.348180 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:27.348127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs\") pod \"network-metrics-daemon-ldqjs\" (UID: \"de44a92d-8440-4b1f-a0ba-479919425485\") " pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:27.348388 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:27.348363    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:27.348462 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:27.348434    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs podName:de44a92d-8440-4b1f-a0ba-479919425485 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:31.348414294 +0000 UTC m=+10.104760107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs") pod "network-metrics-daemon-ldqjs" (UID: "de44a92d-8440-4b1f-a0ba-479919425485") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:27.448949 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:27.448912    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9vcq\" (UniqueName: \"kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq\") pod \"network-check-target-cpmnw\" (UID: \"0f1880b1-1b8e-4633-841a-ed19c927a53d\") " pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:27.449172 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:27.449115    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:27.449172 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:27.449158    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:27.449172 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:27.449172    2560 projected.go:196] Error preparing data for projected volume kube-api-access-v9vcq for pod openshift-network-diagnostics/network-check-target-cpmnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:27.449351 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:27.449236    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq podName:0f1880b1-1b8e-4633-841a-ed19c927a53d nodeName:}" failed. No retries permitted until 2026-07-24 21:15:31.449215755 +0000 UTC m=+10.205561563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9vcq" (UniqueName: "kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq") pod "network-check-target-cpmnw" (UID: "0f1880b1-1b8e-4633-841a-ed19c927a53d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:27.748748 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:27.748714    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:27.748952 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:27.748876    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:28.749503 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:28.749464    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:28.749917 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:28.749592    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:28.749917 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:28.749654    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:28.749917 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:28.749738    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:29.751246 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:29.751214    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:29.751717 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:29.751342    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:30.748516 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:30.748480    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:30.748683 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:30.748527    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:30.748683 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:30.748620    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:30.748798 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:30.748748    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:31.283952 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:31.283906    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:31.284413 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:31.284069    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:31.284413 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:31.284151    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret podName:1ec4ede0-f187-4d6c-a8a2-dd853f431b01 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:39.284116312 +0000 UTC m=+18.040462109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret") pod "global-pull-secret-syncer-b868q" (UID: "1ec4ede0-f187-4d6c-a8a2-dd853f431b01") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:31.384886 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:31.384306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs\") pod \"network-metrics-daemon-ldqjs\" (UID: \"de44a92d-8440-4b1f-a0ba-479919425485\") " pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:31.384886 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:31.384468    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:31.384886 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:31.384535    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs podName:de44a92d-8440-4b1f-a0ba-479919425485 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:39.384515193 +0000 UTC m=+18.140861008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs") pod "network-metrics-daemon-ldqjs" (UID: "de44a92d-8440-4b1f-a0ba-479919425485") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:31.485464 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:31.484891    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9vcq\" (UniqueName: \"kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq\") pod \"network-check-target-cpmnw\" (UID: \"0f1880b1-1b8e-4633-841a-ed19c927a53d\") " pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:31.485464 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:31.485049    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:31.485464 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:31.485067    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:31.485464 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:31.485077    2560 projected.go:196] Error preparing data for projected volume kube-api-access-v9vcq for pod openshift-network-diagnostics/network-check-target-cpmnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:31.485464 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:31.485139    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq podName:0f1880b1-1b8e-4633-841a-ed19c927a53d nodeName:}" failed. No retries permitted until 2026-07-24 21:15:39.485105704 +0000 UTC m=+18.241451496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9vcq" (UniqueName: "kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq") pod "network-check-target-cpmnw" (UID: "0f1880b1-1b8e-4633-841a-ed19c927a53d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:31.751336 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:31.751306    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:31.751517 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:31.751423    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:32.749238 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:32.748647    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:32.749238 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:32.748670    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:32.749238 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:32.748791    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:32.749238 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:32.748890    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:33.751288 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:33.749318    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:33.751288 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:33.749488    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:34.749464 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:34.749279    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:34.749464 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:34.749322    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:34.749464 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:34.749424    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:34.749747 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:34.749547    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:35.749167 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:35.749120    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:35.749641 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:35.749335    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:36.749347 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:36.748900    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:36.749347 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:36.749012    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:36.749347 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:36.749025    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:36.749347 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:36.749163    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:37.749375 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:37.749343    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:37.749812 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:37.749478    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:38.749261 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:38.749218    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:38.749453 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:38.749232    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:38.749453 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:38.749345    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:38.749453 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:38.749426    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:39.347260 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:39.347231    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:39.347424 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:39.347404    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:39.347488 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:39.347480    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret podName:1ec4ede0-f187-4d6c-a8a2-dd853f431b01 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:55.347458554 +0000 UTC m=+34.103804360 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret") pod "global-pull-secret-syncer-b868q" (UID: "1ec4ede0-f187-4d6c-a8a2-dd853f431b01") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:39.448248 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:39.448207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs\") pod \"network-metrics-daemon-ldqjs\" (UID: \"de44a92d-8440-4b1f-a0ba-479919425485\") " pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:39.448421 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:39.448387    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:39.448492 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:39.448478    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs podName:de44a92d-8440-4b1f-a0ba-479919425485 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:55.448457006 +0000 UTC m=+34.204802814 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs") pod "network-metrics-daemon-ldqjs" (UID: "de44a92d-8440-4b1f-a0ba-479919425485") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:39.549468 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:39.549434    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9vcq\" (UniqueName: \"kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq\") pod \"network-check-target-cpmnw\" (UID: \"0f1880b1-1b8e-4633-841a-ed19c927a53d\") " pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:39.549621 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:39.549604    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:39.549695 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:39.549627    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:39.549695 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:39.549637    2560 projected.go:196] Error preparing data for projected volume kube-api-access-v9vcq for pod openshift-network-diagnostics/network-check-target-cpmnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:39.549695 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:39.549694    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq podName:0f1880b1-1b8e-4633-841a-ed19c927a53d nodeName:}" failed. No retries permitted until 2026-07-24 21:15:55.54967472 +0000 UTC m=+34.306020516 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9vcq" (UniqueName: "kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq") pod "network-check-target-cpmnw" (UID: "0f1880b1-1b8e-4633-841a-ed19c927a53d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:39.748708 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:39.748680    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:39.748883 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:39.748817    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:40.749295 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:40.749260    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:40.749746 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:40.749307    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:40.749746 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:40.749404    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:40.749746 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:40.749535    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:41.752048 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:41.752016    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:41.752509 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:41.752158    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:42.749221 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:42.749191    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:42.749385 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:42.749189    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:42.749385 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:42.749297    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:42.749511 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:42.749394    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:43.749066 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:43.748489    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:43.749066 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:43.748822    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:43.961732 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:43.961692    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2qcfz" event={"ID":"e02f6b0b-0d40-4e45-b729-bd64abdb2d64","Type":"ContainerStarted","Data":"396e90e3f987d598d261ff4933a7b15c8e0280935f31f07261a528960e41cafd"}
Jul 24 21:15:43.963515 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:43.963480    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hgz77" event={"ID":"62728d7d-6f8d-43dd-9824-67708026c9ff","Type":"ContainerStarted","Data":"b125d8f4955ca1f2a72d752ad0b8ecf4b8a29b9516ba5ce005f4a1eaec8fec14"}
Jul 24 21:15:43.965071 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:43.965046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p5ggk" event={"ID":"cfd5b405-1f77-48ea-a94f-c30b74583745","Type":"ContainerStarted","Data":"bfdb897f49fccc9915df286cee949a769f6f0e5be8b8c7604c16c3975ee2af60"}
Jul 24 21:15:43.966737 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:43.966712    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5xnz9" event={"ID":"1aafe0ca-0b4a-400d-add6-3998833dd481","Type":"ContainerStarted","Data":"c7d20c59e0d36bfae8050b313279fc5d7c35463501bc8ceb0f3aa862b85442f9"}
Jul 24 21:15:43.968339 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:43.968321    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:15:43.969189 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:43.969170    2560 generic.go:356] "Generic (PLEG): container finished" podID="cb4843e9-e208-4283-bfb9-a5b4799e269f" containerID="fc200ca4e355bb8ce5862ccfdcf0a40e65b5eeaf56321e73bb12bce8aeed6ba0" exitCode=1
Jul 24 21:15:43.969285 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:43.969200    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88v79" event={"ID":"cb4843e9-e208-4283-bfb9-a5b4799e269f","Type":"ContainerDied","Data":"fc200ca4e355bb8ce5862ccfdcf0a40e65b5eeaf56321e73bb12bce8aeed6ba0"}
Jul 24 21:15:43.969344 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:43.969312    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88v79" event={"ID":"cb4843e9-e208-4283-bfb9-a5b4799e269f","Type":"ContainerStarted","Data":"9a7b2a956f8f938f84ee78400de26750d1fd98d57dabd21a2401c5451cb98e8d"}
Jul 24 21:15:43.970779 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:43.970754    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh" event={"ID":"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4","Type":"ContainerStarted","Data":"1ed09ffb6fb7bf37597c4f550f62b2269f9715b52074d123e078e40294ee1b36"}
Jul 24 21:15:43.972216 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:43.972192    2560 generic.go:356] "Generic (PLEG): container finished" podID="a1be7ac6-f32e-40b4-8aa9-abca55380ae4" containerID="f0d78bc1d4442b529b508dc8c01df0132e34acfe8639cf0f9bc21528ed14a1c4" exitCode=0
Jul 24 21:15:43.972306 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:43.972240    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pr7gn" event={"ID":"a1be7ac6-f32e-40b4-8aa9-abca55380ae4","Type":"ContainerDied","Data":"f0d78bc1d4442b529b508dc8c01df0132e34acfe8639cf0f9bc21528ed14a1c4"}
Jul 24 21:15:43.973675 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:43.973663    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p2n8v" event={"ID":"dd8c44fe-e0bf-4ed2-8800-88ccba05358c","Type":"ContainerStarted","Data":"9d9b1b63b5569dc70948590df8d1f93887538229effc9c53db1541102be7d1e3"}
Jul 24 21:15:43.976021 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:43.975977    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-232.ec2.internal" podStartSLOduration=20.975961309 podStartE2EDuration="20.975961309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:26.938912522 +0000 UTC m=+5.695258340" watchObservedRunningTime="2026-07-24 21:15:43.975961309 +0000 UTC m=+22.732307123"
Jul 24 21:15:43.976945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:43.976902    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2qcfz" podStartSLOduration=7.473975128 podStartE2EDuration="21.976886186s" totalImagesPullingTime="14.502911058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:43.976098975 +0000 UTC m=+22.732444771" watchObservedRunningTime="2026-07-24 21:15:43.976886186 +0000 UTC m=+22.733232000"
Jul 24 21:15:44.007225 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.007183    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p5ggk" podStartSLOduration=3.243429438 podStartE2EDuration="22.007167176s" totalImagesPullingTime="18.763737738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:44.007026099 +0000 UTC m=+22.763371925" watchObservedRunningTime="2026-07-24 21:15:44.007167176 +0000 UTC m=+22.763512991"
Jul 24 21:15:44.023076 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.023036    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5xnz9" podStartSLOduration=3.232723678 podStartE2EDuration="22.023023796s" totalImagesPullingTime="18.790300118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:44.022880489 +0000 UTC m=+22.779226303" watchObservedRunningTime="2026-07-24 21:15:44.023023796 +0000 UTC m=+22.779369609"
Jul 24 21:15:44.041141 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.041067    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p2n8v" podStartSLOduration=3.273821957 podStartE2EDuration="22.041055479s" totalImagesPullingTime="18.767233522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:44.040422822 +0000 UTC m=+22.796768637" watchObservedRunningTime="2026-07-24 21:15:44.041055479 +0000 UTC m=+22.797401294"
Jul 24 21:15:44.054731 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.054694    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hgz77" podStartSLOduration=3.290827613 podStartE2EDuration="22.054682282s" totalImagesPullingTime="18.763854669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:44.054382241 +0000 UTC m=+22.810728054" watchObservedRunningTime="2026-07-24 21:15:44.054682282 +0000 UTC m=+22.811028094"
Jul 24 21:15:44.173480 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.173318    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:44.251836 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.251808    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:44.534683 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.534608    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2qcfz"
Jul 24 21:15:44.535276 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.535255    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2qcfz"
Jul 24 21:15:44.749042 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.749017    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:44.749171 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:44.749142    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:44.749698 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.749197    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:44.749698 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:44.749295    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:44.881631 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.881605    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 21:15:44.979067 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.979044    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:15:44.979547 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.979515    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88v79" event={"ID":"cb4843e9-e208-4283-bfb9-a5b4799e269f","Type":"ContainerStarted","Data":"f673a7504fb0ebc086da477d3f083314a7150f08bd78b13a5d40349cb7da8577"}
Jul 24 21:15:44.979685 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.979590    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88v79" event={"ID":"cb4843e9-e208-4283-bfb9-a5b4799e269f","Type":"ContainerStarted","Data":"4876dfb23205656f9db8e044febb0be56e5255df4c35ba1924ab4a41260e6842"}
Jul 24 21:15:44.979685 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.979642    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88v79" event={"ID":"cb4843e9-e208-4283-bfb9-a5b4799e269f","Type":"ContainerStarted","Data":"04adba6bb294cc5741f3726992e331ad58c647dd07bc9cbfe4153bb366d5800e"}
Jul 24 21:15:44.979787 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.979691    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88v79" event={"ID":"cb4843e9-e208-4283-bfb9-a5b4799e269f","Type":"ContainerStarted","Data":"95136b52de6460788ba3aaa2ebf626265d9834df5bd27ce0f24dfbdf6dd3671e"}
Jul 24 21:15:44.981635 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:44.981594    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh" event={"ID":"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4","Type":"ContainerStarted","Data":"e3d057566fc8489ff00b0f611681d34f3c99d414d53fe9d49e75539af53daa24"}
Jul 24 21:15:45.667583 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:45.667439    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T21:15:44.881628969Z","UUID":"6a687854-b86b-4a0b-9b49-d2dd30d9e9f3","Handler":null,"Name":"","Endpoint":""}
Jul 24 21:15:45.669801 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:45.669776    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 21:15:45.669929 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:45.669844    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 21:15:45.751703 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:45.751671    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:45.752059 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:45.751799    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:45.985301 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:45.985207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fchvz" event={"ID":"b1d3a096-c786-43b1-98fd-8e636a4c7e94","Type":"ContainerStarted","Data":"c35c62a7f031172c743a216de4f63b8b1c9852182cf7c9275c772e72068a2b8d"}
Jul 24 21:15:45.987871 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:45.987820    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh" event={"ID":"b3a4ebda-c913-4bd7-8029-5a5a0a0571c4","Type":"ContainerStarted","Data":"e709f617020d3b17fa15d023ff73fa0e5b6b28ae9fbfbeadee290559f1316584"}
Jul 24 21:15:45.987871 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:45.987848    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 21:15:45.999374 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:45.999326    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fchvz" podStartSLOduration=5.241073907 podStartE2EDuration="23.999312098s" totalImagesPullingTime="18.758238191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:45.999204039 +0000 UTC m=+24.755549864" watchObservedRunningTime="2026-07-24 21:15:45.999312098 +0000 UTC m=+24.755657910"
Jul 24 21:15:46.015765 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:46.015729    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-npjbh" podStartSLOduration=3.439254416 podStartE2EDuration="24.015717784s" totalImagesPullingTime="20.576463368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:46.015046009 +0000 UTC m=+24.771391824" watchObservedRunningTime="2026-07-24 21:15:46.015717784 +0000 UTC m=+24.772063597"
Jul 24 21:15:46.121719 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:46.121690    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:46.176228 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:46.176187    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:46.748492 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:46.748415    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:46.748492 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:46.748447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:46.748701 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:46.748532    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:46.751926 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:46.749117    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:47.749042 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:47.748872    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:47.749255 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:47.749177    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:47.996468 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:47.996444    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:15:47.996961 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:47.996795    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88v79" event={"ID":"cb4843e9-e208-4283-bfb9-a5b4799e269f","Type":"ContainerStarted","Data":"c6a45aabff47150314c16280b5113f1f6718de0ebfda36267d568004fbc7d10a"}
Jul 24 21:15:48.748445 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:48.748409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:48.748605 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:48.748409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:48.748605 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:48.748542    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:48.748744 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:48.748609    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:49.000299 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:49.000226    2560 generic.go:356] "Generic (PLEG): container finished" podID="a1be7ac6-f32e-40b4-8aa9-abca55380ae4" containerID="13effe53e164ef2de009e443216a57730fd2d7abc426869e4f2e06e137db3aa1" exitCode=0
Jul 24 21:15:49.000299 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:49.000269    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pr7gn" event={"ID":"a1be7ac6-f32e-40b4-8aa9-abca55380ae4","Type":"ContainerDied","Data":"13effe53e164ef2de009e443216a57730fd2d7abc426869e4f2e06e137db3aa1"}
Jul 24 21:15:49.749009 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:49.748988    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:49.749114 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:49.749093    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:50.008034 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:50.008003    2560 generic.go:356] "Generic (PLEG): container finished" podID="a1be7ac6-f32e-40b4-8aa9-abca55380ae4" containerID="6de7b817fcfb8388ec02813db3962d58c79ba378e36d3c013b238cdbd19308f1" exitCode=0
Jul 24 21:15:50.008400 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:50.008079    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pr7gn" event={"ID":"a1be7ac6-f32e-40b4-8aa9-abca55380ae4","Type":"ContainerDied","Data":"6de7b817fcfb8388ec02813db3962d58c79ba378e36d3c013b238cdbd19308f1"}
Jul 24 21:15:50.749689 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:50.749481    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:50.749774 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:50.749516    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:50.749835 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:50.749812    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:50.749889 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:50.749870    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:51.013476 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:51.013401    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:15:51.013828 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:51.013803    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88v79" event={"ID":"cb4843e9-e208-4283-bfb9-a5b4799e269f","Type":"ContainerStarted","Data":"d19e015c13cfba0c915326f15f315c01bcd6b5665884a589da637aed51bd5361"}
Jul 24 21:15:51.014077 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:51.014053    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:51.014300 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:51.014280    2560 scope.go:117] "RemoveContainer" containerID="fc200ca4e355bb8ce5862ccfdcf0a40e65b5eeaf56321e73bb12bce8aeed6ba0"
Jul 24 21:15:51.016557 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:51.016531    2560 generic.go:356] "Generic (PLEG): container finished" podID="a1be7ac6-f32e-40b4-8aa9-abca55380ae4" containerID="a66a0bf020a6ec5bcb0c01c95912cf71d49e490eeb220bff5a74c8c2e7d8baf8" exitCode=0
Jul 24 21:15:51.016644 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:51.016572    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pr7gn" event={"ID":"a1be7ac6-f32e-40b4-8aa9-abca55380ae4","Type":"ContainerDied","Data":"a66a0bf020a6ec5bcb0c01c95912cf71d49e490eeb220bff5a74c8c2e7d8baf8"}
Jul 24 21:15:51.030966 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:51.030945    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:51.738952 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:51.737786    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2qcfz"
Jul 24 21:15:51.738952 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:51.738156    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 24 21:15:51.738952 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:51.738721    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2qcfz"
Jul 24 21:15:51.749484 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:51.749461    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:51.749614 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:51.749580    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:51.813413 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:51.813199    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:52.013538 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:52.013469    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cpmnw"]
Jul 24 21:15:52.013976 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:52.013577    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:52.013976 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:52.013668    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:52.017691 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:52.016156    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b868q"]
Jul 24 21:15:52.017691 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:52.016215    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldqjs"]
Jul 24 21:15:52.017691 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:52.016298    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:52.017691 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:52.016385    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:52.027990 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:52.027970    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:15:52.030060 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:52.030036    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:52.030204 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:52.030072    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-88v79" event={"ID":"cb4843e9-e208-4283-bfb9-a5b4799e269f","Type":"ContainerStarted","Data":"9f9150366e6369375c886fd58a20765c93f941ee622cda5fa412b0b34ae86c88"}
Jul 24 21:15:52.030281 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:52.030224    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:52.030804 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:52.030662    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:52.030804 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:52.030741    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:52.050547 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:52.050518    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:15:52.112208 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:52.112159    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-88v79" podStartSLOduration=11.273866197 podStartE2EDuration="30.11212078s" totalImagesPullingTime="18.838254583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:52.08365599 +0000 UTC m=+30.840001805" watchObservedRunningTime="2026-07-24 21:15:52.11212078 +0000 UTC m=+30.868466593"
Jul 24 21:15:53.749487 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:53.749450    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:53.749487 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:53.749505    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:53.750056 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:53.749450    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:53.750056 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:53.749586    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:53.750056 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:53.749679    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:53.750056 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:53.749737    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:55.368341 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:55.368306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:55.368873 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:55.368464    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:55.368873 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:55.368526    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret podName:1ec4ede0-f187-4d6c-a8a2-dd853f431b01 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:27.368510409 +0000 UTC m=+66.124856210 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret") pod "global-pull-secret-syncer-b868q" (UID: "1ec4ede0-f187-4d6c-a8a2-dd853f431b01") : object "kube-system"/"original-pull-secret" not registered
Jul 24 21:15:55.469416 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:55.469383    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs\") pod \"network-metrics-daemon-ldqjs\" (UID: \"de44a92d-8440-4b1f-a0ba-479919425485\") " pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:55.469596 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:55.469522    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:55.469596 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:55.469575    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs podName:de44a92d-8440-4b1f-a0ba-479919425485 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:27.469560426 +0000 UTC m=+66.225906216 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs") pod "network-metrics-daemon-ldqjs" (UID: "de44a92d-8440-4b1f-a0ba-479919425485") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 21:15:55.570226 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:55.570196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9vcq\" (UniqueName: \"kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq\") pod \"network-check-target-cpmnw\" (UID: \"0f1880b1-1b8e-4633-841a-ed19c927a53d\") " pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:55.570342 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:55.570322    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 21:15:55.570394 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:55.570351    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 21:15:55.570394 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:55.570365    2560 projected.go:196] Error preparing data for projected volume kube-api-access-v9vcq for pod openshift-network-diagnostics/network-check-target-cpmnw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:55.570470 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:55.570431    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq podName:0f1880b1-1b8e-4633-841a-ed19c927a53d nodeName:}" failed. No retries permitted until 2026-07-24 21:16:27.570411467 +0000 UTC m=+66.326757273 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9vcq" (UniqueName: "kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq") pod "network-check-target-cpmnw" (UID: "0f1880b1-1b8e-4633-841a-ed19c927a53d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 21:15:55.748889 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:55.748859    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:55.748889 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:55.748876    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:55.749074 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:55.748869    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:55.749074 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:55.748979    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b868q" podUID="1ec4ede0-f187-4d6c-a8a2-dd853f431b01"
Jul 24 21:15:55.749074 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:55.749026    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldqjs" podUID="de44a92d-8440-4b1f-a0ba-479919425485"
Jul 24 21:15:55.749245 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:55.749082    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cpmnw" podUID="0f1880b1-1b8e-4633-841a-ed19c927a53d"
Jul 24 21:15:56.042865 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.042779    2560 generic.go:356] "Generic (PLEG): container finished" podID="a1be7ac6-f32e-40b4-8aa9-abca55380ae4" containerID="ab86d1b4db209a058b2dd30e82dd54017150a15ccd09d7b8e6e0c5e4e946cc88" exitCode=0
Jul 24 21:15:56.043000 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.042864    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pr7gn" event={"ID":"a1be7ac6-f32e-40b4-8aa9-abca55380ae4","Type":"ContainerDied","Data":"ab86d1b4db209a058b2dd30e82dd54017150a15ccd09d7b8e6e0c5e4e946cc88"}
Jul 24 21:15:56.528409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.528378    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-232.ec2.internal" event="NodeReady"
Jul 24 21:15:56.528894 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.528513    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 21:15:56.579318 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.579295    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hzssv"]
Jul 24 21:15:56.597540 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.597515    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j8cqc"]
Jul 24 21:15:56.597695 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.597635    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hzssv"
Jul 24 21:15:56.600614 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.600595    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vk68f\""
Jul 24 21:15:56.600871 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.600858    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 21:15:56.600924 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.600862    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 21:15:56.600924 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.600918    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 21:15:56.613596 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.613576    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hzssv"]
Jul 24 21:15:56.613697 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.613607    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j8cqc"]
Jul 24 21:15:56.613697 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.613684    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:15:56.616405 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.616387    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 21:15:56.617079 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.616923    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 21:15:56.617079 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.616977    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-thbdw\""
Jul 24 21:15:56.731510 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.731483    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:56.780714 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.780657    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb92e8b5-ef73-45bc-b383-718ecad57138-cert\") pod \"ingress-canary-hzssv\" (UID: \"fb92e8b5-ef73-45bc-b383-718ecad57138\") " pod="openshift-ingress-canary/ingress-canary-hzssv"
Jul 24 21:15:56.780714 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.780690    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m82d\" (UniqueName: \"kubernetes.io/projected/fb92e8b5-ef73-45bc-b383-718ecad57138-kube-api-access-9m82d\") pod \"ingress-canary-hzssv\" (UID: \"fb92e8b5-ef73-45bc-b383-718ecad57138\") " pod="openshift-ingress-canary/ingress-canary-hzssv"
Jul 24 21:15:56.780844 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.780711    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ade7d3-1225-4c63-8282-4b0ea5537005-config-volume\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:15:56.780844 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.780827    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0ade7d3-1225-4c63-8282-4b0ea5537005-tmp-dir\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:15:56.780912 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.780878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:15:56.780944 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.780910    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pp5p\" (UniqueName: \"kubernetes.io/projected/b0ade7d3-1225-4c63-8282-4b0ea5537005-kube-api-access-6pp5p\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:15:56.802437 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.802408    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:56.881533 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.881508    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb92e8b5-ef73-45bc-b383-718ecad57138-cert\") pod \"ingress-canary-hzssv\" (UID: \"fb92e8b5-ef73-45bc-b383-718ecad57138\") " pod="openshift-ingress-canary/ingress-canary-hzssv"
Jul 24 21:15:56.881669 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.881541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m82d\" (UniqueName: \"kubernetes.io/projected/fb92e8b5-ef73-45bc-b383-718ecad57138-kube-api-access-9m82d\") pod \"ingress-canary-hzssv\" (UID: \"fb92e8b5-ef73-45bc-b383-718ecad57138\") " pod="openshift-ingress-canary/ingress-canary-hzssv"
Jul 24 21:15:56.881669 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.881559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ade7d3-1225-4c63-8282-4b0ea5537005-config-volume\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:15:56.881669 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.881590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0ade7d3-1225-4c63-8282-4b0ea5537005-tmp-dir\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:15:56.881838 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.881782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:15:56.881838 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.881819    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pp5p\" (UniqueName: \"kubernetes.io/projected/b0ade7d3-1225-4c63-8282-4b0ea5537005-kube-api-access-6pp5p\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:15:56.881923 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.881892    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0ade7d3-1225-4c63-8282-4b0ea5537005-tmp-dir\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:15:56.881971 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:56.881918    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:15:56.882018 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:56.881979    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls podName:b0ade7d3-1225-4c63-8282-4b0ea5537005 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:57.381961192 +0000 UTC m=+36.138306999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls") pod "dns-default-j8cqc" (UID: "b0ade7d3-1225-4c63-8282-4b0ea5537005") : secret "dns-default-metrics-tls" not found
Jul 24 21:15:56.882249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.882228    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ade7d3-1225-4c63-8282-4b0ea5537005-config-volume\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:15:56.885976 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.885956    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb92e8b5-ef73-45bc-b383-718ecad57138-cert\") pod \"ingress-canary-hzssv\" (UID: \"fb92e8b5-ef73-45bc-b383-718ecad57138\") " pod="openshift-ingress-canary/ingress-canary-hzssv"
Jul 24 21:15:56.899028 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.899008    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pp5p\" (UniqueName: \"kubernetes.io/projected/b0ade7d3-1225-4c63-8282-4b0ea5537005-kube-api-access-6pp5p\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:15:56.899168 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.899118    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m82d\" (UniqueName: \"kubernetes.io/projected/fb92e8b5-ef73-45bc-b383-718ecad57138-kube-api-access-9m82d\") pod \"ingress-canary-hzssv\" (UID: \"fb92e8b5-ef73-45bc-b383-718ecad57138\") " pod="openshift-ingress-canary/ingress-canary-hzssv"
Jul 24 21:15:56.906916 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:56.906891    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hzssv"
Jul 24 21:15:57.049024 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:57.048750    2560 generic.go:356] "Generic (PLEG): container finished" podID="a1be7ac6-f32e-40b4-8aa9-abca55380ae4" containerID="3318fc159dc10357bf3ef59b3fa5d69d34698f276313231bc9ffe537a8cbe247" exitCode=0
Jul 24 21:15:57.049024 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:57.048814    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pr7gn" event={"ID":"a1be7ac6-f32e-40b4-8aa9-abca55380ae4","Type":"ContainerDied","Data":"3318fc159dc10357bf3ef59b3fa5d69d34698f276313231bc9ffe537a8cbe247"}
Jul 24 21:15:57.104888 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:57.104858    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hzssv"]
Jul 24 21:15:57.109046 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:15:57.109020    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb92e8b5_ef73_45bc_b383_718ecad57138.slice/crio-3c4a4dddfcda771484b395d7796ec1dd4934d3568eb19002a017df16ac9514df WatchSource:0}: Error finding container 3c4a4dddfcda771484b395d7796ec1dd4934d3568eb19002a017df16ac9514df: Status 404 returned error can't find the container with id 3c4a4dddfcda771484b395d7796ec1dd4934d3568eb19002a017df16ac9514df
Jul 24 21:15:57.385931 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:57.385896    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:15:57.386087 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:57.386048    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:15:57.386152 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:57.386113    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls podName:b0ade7d3-1225-4c63-8282-4b0ea5537005 nodeName:}" failed. No retries permitted until 2026-07-24 21:15:58.386097737 +0000 UTC m=+37.142443527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls") pod "dns-default-j8cqc" (UID: "b0ade7d3-1225-4c63-8282-4b0ea5537005") : secret "dns-default-metrics-tls" not found
Jul 24 21:15:57.749236 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:57.749064    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:15:57.749236 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:57.749064    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:15:57.749809 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:57.749064    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:15:57.755870 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:57.755848    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 21:15:57.756015 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:57.755887    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n9q4d\""
Jul 24 21:15:57.756015 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:57.755857    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 21:15:57.757148 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:57.757110    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdtkv\""
Jul 24 21:15:57.758214 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:57.757487    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 21:15:57.758214 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:57.757556    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 21:15:57.817469 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:57.817439    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:57.881501 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:57.881467    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:58.052749 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:58.052660    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hzssv" event={"ID":"fb92e8b5-ef73-45bc-b383-718ecad57138","Type":"ContainerStarted","Data":"3c4a4dddfcda771484b395d7796ec1dd4934d3568eb19002a017df16ac9514df"}
Jul 24 21:15:58.056885 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:58.056805    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pr7gn" event={"ID":"a1be7ac6-f32e-40b4-8aa9-abca55380ae4","Type":"ContainerStarted","Data":"2e734e7f4f76c7fe00e367f760f59525438e19a979d7f01208672838303a6ba0"}
Jul 24 21:15:58.081400 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:58.081351    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pr7gn" podStartSLOduration=7.537634244 podStartE2EDuration="36.081335772s" totalImagesPullingTime="28.543701528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:15:58.081158014 +0000 UTC m=+36.837503827" watchObservedRunningTime="2026-07-24 21:15:58.081335772 +0000 UTC m=+36.837681585"
Jul 24 21:15:58.206729 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:58.206699    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:58.309469 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:58.309386    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:15:58.396095 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:15:58.396054    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:15:58.396286 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:58.396222    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:15:58.396339 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:15:58.396289    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls podName:b0ade7d3-1225-4c63-8282-4b0ea5537005 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:00.396272869 +0000 UTC m=+39.152618663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls") pod "dns-default-j8cqc" (UID: "b0ade7d3-1225-4c63-8282-4b0ea5537005") : secret "dns-default-metrics-tls" not found
Jul 24 21:16:00.063216 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:00.062942    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hzssv" event={"ID":"fb92e8b5-ef73-45bc-b383-718ecad57138","Type":"ContainerStarted","Data":"a64af758417f3a62dc829fb3292df8e5380f198cfd18f362178fd7c6f79dfd19"}
Jul 24 21:16:00.081017 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:00.080970    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hzssv" podStartSLOduration=1.5999754 podStartE2EDuration="4.080954144s" totalImagesPullingTime="2.480978744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:00.080184937 +0000 UTC m=+38.836530751" watchObservedRunningTime="2026-07-24 21:16:00.080954144 +0000 UTC m=+38.837299935"
Jul 24 21:16:00.410447 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:00.410416    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:16:00.410599 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:00.410565    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:16:00.410641 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:00.410629    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls podName:b0ade7d3-1225-4c63-8282-4b0ea5537005 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:04.410613983 +0000 UTC m=+43.166959773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls") pod "dns-default-j8cqc" (UID: "b0ade7d3-1225-4c63-8282-4b0ea5537005") : secret "dns-default-metrics-tls" not found
Jul 24 21:16:01.605188 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:01.605160    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:02.212831 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.212790    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"]
Jul 24 21:16:02.241938 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.241916    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"]
Jul 24 21:16:02.242080 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.242001    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:02.248762 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.248737    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 21:16:02.248882 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.248845    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 21:16:02.248937 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.248913    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 24 21:16:02.249010 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.248994    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 24 21:16:02.249068 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.249052    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mrlmd\""
Jul 24 21:16:02.423339 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.423304    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vlf8v\" (UID: \"c944d753-b675-4664-944a-3ec3c10b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:02.423339 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.423362    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c944d753-b675-4664-944a-3ec3c10b224f-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-vlf8v\" (UID: \"c944d753-b675-4664-944a-3ec3c10b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:02.423548 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.423401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfgm7\" (UniqueName: \"kubernetes.io/projected/c944d753-b675-4664-944a-3ec3c10b224f-kube-api-access-zfgm7\") pod \"cluster-monitoring-operator-548b489cb8-vlf8v\" (UID: \"c944d753-b675-4664-944a-3ec3c10b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:02.483507 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.483438    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:02.524544 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.524518    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vlf8v\" (UID: \"c944d753-b675-4664-944a-3ec3c10b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:02.524668 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.524573    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c944d753-b675-4664-944a-3ec3c10b224f-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-vlf8v\" (UID: \"c944d753-b675-4664-944a-3ec3c10b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:02.524668 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.524614    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfgm7\" (UniqueName: \"kubernetes.io/projected/c944d753-b675-4664-944a-3ec3c10b224f-kube-api-access-zfgm7\") pod \"cluster-monitoring-operator-548b489cb8-vlf8v\" (UID: \"c944d753-b675-4664-944a-3ec3c10b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:02.524668 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:02.524657    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 21:16:02.524813 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:02.524724    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls podName:c944d753-b675-4664-944a-3ec3c10b224f nodeName:}" failed. No retries permitted until 2026-07-24 21:16:03.024703254 +0000 UTC m=+41.781049044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vlf8v" (UID: "c944d753-b675-4664-944a-3ec3c10b224f") : secret "cluster-monitoring-operator-tls" not found
Jul 24 21:16:02.525336 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.525315    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c944d753-b675-4664-944a-3ec3c10b224f-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-vlf8v\" (UID: \"c944d753-b675-4664-944a-3ec3c10b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:02.539300 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.539280    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfgm7\" (UniqueName: \"kubernetes.io/projected/c944d753-b675-4664-944a-3ec3c10b224f-kube-api-access-zfgm7\") pod \"cluster-monitoring-operator-548b489cb8-vlf8v\" (UID: \"c944d753-b675-4664-944a-3ec3c10b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:02.584998 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:02.584971    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:03.027217 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:03.027179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vlf8v\" (UID: \"c944d753-b675-4664-944a-3ec3c10b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:03.027593 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:03.027297    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 21:16:03.027593 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:03.027350    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls podName:c944d753-b675-4664-944a-3ec3c10b224f nodeName:}" failed. No retries permitted until 2026-07-24 21:16:04.02733533 +0000 UTC m=+42.783681121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vlf8v" (UID: "c944d753-b675-4664-944a-3ec3c10b224f") : secret "cluster-monitoring-operator-tls" not found
Jul 24 21:16:04.031943 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:04.031905    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vlf8v\" (UID: \"c944d753-b675-4664-944a-3ec3c10b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:04.032340 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:04.032060    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 21:16:04.032340 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:04.032144    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls podName:c944d753-b675-4664-944a-3ec3c10b224f nodeName:}" failed. No retries permitted until 2026-07-24 21:16:06.032109969 +0000 UTC m=+44.788455759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vlf8v" (UID: "c944d753-b675-4664-944a-3ec3c10b224f") : secret "cluster-monitoring-operator-tls" not found
Jul 24 21:16:04.276671 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:04.276640    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:04.380364 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:04.380335    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:04.435207 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:04.435187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:16:04.435350 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:04.435334    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:16:04.435403 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:04.435394    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls podName:b0ade7d3-1225-4c63-8282-4b0ea5537005 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:12.435378704 +0000 UTC m=+51.191724520 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls") pod "dns-default-j8cqc" (UID: "b0ade7d3-1225-4c63-8282-4b0ea5537005") : secret "dns-default-metrics-tls" not found
Jul 24 21:16:05.692662 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.692631    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r75fx"]
Jul 24 21:16:05.724276 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.724254    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:05.729515 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.729501    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r75fx"]
Jul 24 21:16:05.729594 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.729589    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:05.731841 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.731821    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 21:16:05.731949 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.731851    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 21:16:05.732673 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.732655    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 21:16:05.732752 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.732660    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 21:16:05.732752 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.732663    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4d5n4\""
Jul 24 21:16:05.792643 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.792615    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:05.846747 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.846719    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8999bbe-7e48-4678-8951-b232a39f1de9-data-volume\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:05.846865 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.846754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2h7l\" (UniqueName: \"kubernetes.io/projected/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-api-access-m2h7l\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:05.846865 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.846786    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8999bbe-7e48-4678-8951-b232a39f1de9-crio-socket\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:05.846865 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.846821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:05.846865 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.846860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:05.948198 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.948091    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8999bbe-7e48-4678-8951-b232a39f1de9-data-volume\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:05.948198 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.948148    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2h7l\" (UniqueName: \"kubernetes.io/projected/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-api-access-m2h7l\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:05.948396 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.948265    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8999bbe-7e48-4678-8951-b232a39f1de9-crio-socket\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:05.948396 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.948308    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:05.948396 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.948339    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:05.948550 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.948460    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d8999bbe-7e48-4678-8951-b232a39f1de9-crio-socket\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:05.948550 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:05.948480    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 21:16:05.948550 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:05.948508    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:05.948550 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.948501    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d8999bbe-7e48-4678-8951-b232a39f1de9-data-volume\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:05.948693 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:05.948583    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm podName:d8999bbe-7e48-4678-8951-b232a39f1de9 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:06.448563434 +0000 UTC m=+45.204909247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r75fx" (UID: "d8999bbe-7e48-4678-8951-b232a39f1de9") : configmap "kube-rbac-proxy" not found
Jul 24 21:16:05.948693 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:05.948617    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls podName:d8999bbe-7e48-4678-8951-b232a39f1de9 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:06.448594304 +0000 UTC m=+45.204940107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r75fx" (UID: "d8999bbe-7e48-4678-8951-b232a39f1de9") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:05.957281 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:05.957260    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2h7l\" (UniqueName: \"kubernetes.io/projected/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-api-access-m2h7l\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:06.049170 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:06.049118    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vlf8v\" (UID: \"c944d753-b675-4664-944a-3ec3c10b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:06.049335 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:06.049262    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 21:16:06.049335 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:06.049325    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls podName:c944d753-b675-4664-944a-3ec3c10b224f nodeName:}" failed. No retries permitted until 2026-07-24 21:16:10.049311374 +0000 UTC m=+48.805657165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vlf8v" (UID: "c944d753-b675-4664-944a-3ec3c10b224f") : secret "cluster-monitoring-operator-tls" not found
Jul 24 21:16:06.451753 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:06.451715    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:06.451896 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:06.451848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:06.451896 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:06.451878    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:06.451987 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:06.451916    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 21:16:06.451987 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:06.451949    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls podName:d8999bbe-7e48-4678-8951-b232a39f1de9 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:07.451930289 +0000 UTC m=+46.208276095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r75fx" (UID: "d8999bbe-7e48-4678-8951-b232a39f1de9") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:06.451987 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:06.451966    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm podName:d8999bbe-7e48-4678-8951-b232a39f1de9 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:07.451955896 +0000 UTC m=+46.208301688 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r75fx" (UID: "d8999bbe-7e48-4678-8951-b232a39f1de9") : configmap "kube-rbac-proxy" not found
Jul 24 21:16:07.457966 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:07.457927    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:07.458419 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:07.458030    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:07.458419 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:07.458106    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 21:16:07.458419 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:07.458115    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:07.458419 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:07.458175    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm podName:d8999bbe-7e48-4678-8951-b232a39f1de9 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:09.458162041 +0000 UTC m=+48.214507845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r75fx" (UID: "d8999bbe-7e48-4678-8951-b232a39f1de9") : configmap "kube-rbac-proxy" not found
Jul 24 21:16:07.458419 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:07.458202    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls podName:d8999bbe-7e48-4678-8951-b232a39f1de9 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:09.458182658 +0000 UTC m=+48.214528450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r75fx" (UID: "d8999bbe-7e48-4678-8951-b232a39f1de9") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:07.637056 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:07.637016    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hgz77_62728d7d-6f8d-43dd-9824-67708026c9ff/dns-node-resolver/0.log"
Jul 24 21:16:08.636731 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:08.636702    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p5ggk_cfd5b405-1f77-48ea-a94f-c30b74583745/node-ca/0.log"
Jul 24 21:16:09.236583 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:09.236555    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hzssv_fb92e8b5-ef73-45bc-b383-718ecad57138/serve-healthcheck-canary/0.log"
Jul 24 21:16:09.473009 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:09.472970    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:09.473216 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:09.473022    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:09.473216 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:09.473094    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 21:16:09.473216 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:09.473163    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:09.473216 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:09.473183    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm podName:d8999bbe-7e48-4678-8951-b232a39f1de9 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:13.473162799 +0000 UTC m=+52.229508609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r75fx" (UID: "d8999bbe-7e48-4678-8951-b232a39f1de9") : configmap "kube-rbac-proxy" not found
Jul 24 21:16:09.473216 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:09.473215    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls podName:d8999bbe-7e48-4678-8951-b232a39f1de9 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:13.473200896 +0000 UTC m=+52.229546713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r75fx" (UID: "d8999bbe-7e48-4678-8951-b232a39f1de9") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:10.076710 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:10.076680    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vlf8v\" (UID: \"c944d753-b675-4664-944a-3ec3c10b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:10.077090 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:10.076843    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 21:16:10.077090 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:10.076911    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls podName:c944d753-b675-4664-944a-3ec3c10b224f nodeName:}" failed. No retries permitted until 2026-07-24 21:16:18.076893382 +0000 UTC m=+56.833239193 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vlf8v" (UID: "c944d753-b675-4664-944a-3ec3c10b224f") : secret "cluster-monitoring-operator-tls" not found
Jul 24 21:16:12.229341 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.229309    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj"]
Jul 24 21:16:12.246194 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.246168    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ppqph"]
Jul 24 21:16:12.246335 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.246280    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj"
Jul 24 21:16:12.249292 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.249267    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 24 21:16:12.249415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.249394    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6bsqq\""
Jul 24 21:16:12.249478 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.249447    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 24 21:16:12.249676 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.249657    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 24 21:16:12.249935 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.249923    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 24 21:16:12.260168 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.260150    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6"]
Jul 24 21:16:12.260275 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.260261    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ppqph"
Jul 24 21:16:12.262526 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.262502    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 24 21:16:12.262681 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.262668    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 24 21:16:12.262770 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.262754    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jw8w9\""
Jul 24 21:16:12.278309 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.278289    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"]
Jul 24 21:16:12.278412 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.278398    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6"
Jul 24 21:16:12.280955 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.280938    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 24 21:16:12.281251 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.281237    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xbg7b\""
Jul 24 21:16:12.281472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.281456    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 24 21:16:12.281554 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.281493    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 24 21:16:12.282027 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.282015    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 24 21:16:12.292216 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.292194    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6v64\" (UniqueName: \"kubernetes.io/projected/38bf3087-9101-4b71-be34-dd0726961e2a-kube-api-access-r6v64\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6\" (UID: \"38bf3087-9101-4b71-be34-dd0726961e2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6"
Jul 24 21:16:12.292297 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.292225    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pknbs\" (UniqueName: \"kubernetes.io/projected/bb8acf81-3fb0-4095-9ae4-b9b514ffffe1-kube-api-access-pknbs\") pod \"volume-data-source-validator-549c6d54cf-ppqph\" (UID: \"bb8acf81-3fb0-4095-9ae4-b9b514ffffe1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ppqph"
Jul 24 21:16:12.292354 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.292339    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38bf3087-9101-4b71-be34-dd0726961e2a-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6\" (UID: \"38bf3087-9101-4b71-be34-dd0726961e2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6"
Jul 24 21:16:12.292393 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.292366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38bf3087-9101-4b71-be34-dd0726961e2a-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6\" (UID: \"38bf3087-9101-4b71-be34-dd0726961e2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6"
Jul 24 21:16:12.292393 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.292383    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgz92\" (UniqueName: \"kubernetes.io/projected/df093810-8a45-4cc0-b0e2-a1839e46b254-kube-api-access-kgz92\") pod \"service-ca-operator-5d6c75f84f-2ffrj\" (UID: \"df093810-8a45-4cc0-b0e2-a1839e46b254\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj"
Jul 24 21:16:12.292453 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.292424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df093810-8a45-4cc0-b0e2-a1839e46b254-serving-cert\") pod \"service-ca-operator-5d6c75f84f-2ffrj\" (UID: \"df093810-8a45-4cc0-b0e2-a1839e46b254\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj"
Jul 24 21:16:12.292485 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.292466    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df093810-8a45-4cc0-b0e2-a1839e46b254-config\") pod \"service-ca-operator-5d6c75f84f-2ffrj\" (UID: \"df093810-8a45-4cc0-b0e2-a1839e46b254\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj"
Jul 24 21:16:12.304529 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.304510    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6"]
Jul 24 21:16:12.304614 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.304535    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj"]
Jul 24 21:16:12.304614 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.304548    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"]
Jul 24 21:16:12.304614 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.304563    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ppqph"]
Jul 24 21:16:12.304705 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.304621    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"
Jul 24 21:16:12.309013 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.308996    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 24 21:16:12.309378 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.309363    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 24 21:16:12.309469 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.309383    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 24 21:16:12.309469 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.309384    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6qmgl\""
Jul 24 21:16:12.393640 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.393612    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6v64\" (UniqueName: \"kubernetes.io/projected/38bf3087-9101-4b71-be34-dd0726961e2a-kube-api-access-r6v64\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6\" (UID: \"38bf3087-9101-4b71-be34-dd0726961e2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6"
Jul 24 21:16:12.393748 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.393644    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6cp2\" (UniqueName: \"kubernetes.io/projected/03156781-088a-4815-bcfd-cdc309e27c13-kube-api-access-r6cp2\") pod \"cluster-samples-operator-6748489656-fxwlp\" (UID: \"03156781-088a-4815-bcfd-cdc309e27c13\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"
Jul 24 21:16:12.393748 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.393666    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pknbs\" (UniqueName: \"kubernetes.io/projected/bb8acf81-3fb0-4095-9ae4-b9b514ffffe1-kube-api-access-pknbs\") pod \"volume-data-source-validator-549c6d54cf-ppqph\" (UID: \"bb8acf81-3fb0-4095-9ae4-b9b514ffffe1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ppqph"
Jul 24 21:16:12.393748 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.393685    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-fxwlp\" (UID: \"03156781-088a-4815-bcfd-cdc309e27c13\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"
Jul 24 21:16:12.393888 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.393871    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38bf3087-9101-4b71-be34-dd0726961e2a-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6\" (UID: \"38bf3087-9101-4b71-be34-dd0726961e2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6"
Jul 24 21:16:12.393926 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.393907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38bf3087-9101-4b71-be34-dd0726961e2a-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6\" (UID: \"38bf3087-9101-4b71-be34-dd0726961e2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6"
Jul 24 21:16:12.393960 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.393926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgz92\" (UniqueName: \"kubernetes.io/projected/df093810-8a45-4cc0-b0e2-a1839e46b254-kube-api-access-kgz92\") pod \"service-ca-operator-5d6c75f84f-2ffrj\" (UID: \"df093810-8a45-4cc0-b0e2-a1839e46b254\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj"
Jul 24 21:16:12.394072 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.394055    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df093810-8a45-4cc0-b0e2-a1839e46b254-serving-cert\") pod \"service-ca-operator-5d6c75f84f-2ffrj\" (UID: \"df093810-8a45-4cc0-b0e2-a1839e46b254\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj"
Jul 24 21:16:12.394158 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.394120    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df093810-8a45-4cc0-b0e2-a1839e46b254-config\") pod \"service-ca-operator-5d6c75f84f-2ffrj\" (UID: \"df093810-8a45-4cc0-b0e2-a1839e46b254\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj"
Jul 24 21:16:12.394555 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.394526    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38bf3087-9101-4b71-be34-dd0726961e2a-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6\" (UID: \"38bf3087-9101-4b71-be34-dd0726961e2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6"
Jul 24 21:16:12.394643 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.394625    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df093810-8a45-4cc0-b0e2-a1839e46b254-config\") pod \"service-ca-operator-5d6c75f84f-2ffrj\" (UID: \"df093810-8a45-4cc0-b0e2-a1839e46b254\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj"
Jul 24 21:16:12.396171 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.396155    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38bf3087-9101-4b71-be34-dd0726961e2a-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6\" (UID: \"38bf3087-9101-4b71-be34-dd0726961e2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6"
Jul 24 21:16:12.396229 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.396207    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/df093810-8a45-4cc0-b0e2-a1839e46b254-serving-cert\") pod \"service-ca-operator-5d6c75f84f-2ffrj\" (UID: \"df093810-8a45-4cc0-b0e2-a1839e46b254\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj"
Jul 24 21:16:12.402586 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.402564    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6v64\" (UniqueName: \"kubernetes.io/projected/38bf3087-9101-4b71-be34-dd0726961e2a-kube-api-access-r6v64\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6\" (UID: \"38bf3087-9101-4b71-be34-dd0726961e2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6"
Jul 24 21:16:12.402870 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.402852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgz92\" (UniqueName: \"kubernetes.io/projected/df093810-8a45-4cc0-b0e2-a1839e46b254-kube-api-access-kgz92\") pod \"service-ca-operator-5d6c75f84f-2ffrj\" (UID: \"df093810-8a45-4cc0-b0e2-a1839e46b254\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj"
Jul 24 21:16:12.404445 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.404427    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pknbs\" (UniqueName: \"kubernetes.io/projected/bb8acf81-3fb0-4095-9ae4-b9b514ffffe1-kube-api-access-pknbs\") pod \"volume-data-source-validator-549c6d54cf-ppqph\" (UID: \"bb8acf81-3fb0-4095-9ae4-b9b514ffffe1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ppqph"
Jul 24 21:16:12.495178 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.495103    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6cp2\" (UniqueName: \"kubernetes.io/projected/03156781-088a-4815-bcfd-cdc309e27c13-kube-api-access-r6cp2\") pod \"cluster-samples-operator-6748489656-fxwlp\" (UID: \"03156781-088a-4815-bcfd-cdc309e27c13\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"
Jul 24 21:16:12.495178 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.495154    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-fxwlp\" (UID: \"03156781-088a-4815-bcfd-cdc309e27c13\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"
Jul 24 21:16:12.495370 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.495206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:16:12.495370 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:12.495303    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 21:16:12.495370 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:12.495317    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 21:16:12.495370 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:12.495356    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls podName:b0ade7d3-1225-4c63-8282-4b0ea5537005 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:28.49534328 +0000 UTC m=+67.251689072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls") pod "dns-default-j8cqc" (UID: "b0ade7d3-1225-4c63-8282-4b0ea5537005") : secret "dns-default-metrics-tls" not found
Jul 24 21:16:12.495370 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:12.495368    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls podName:03156781-088a-4815-bcfd-cdc309e27c13 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:12.995362353 +0000 UTC m=+51.751708144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls") pod "cluster-samples-operator-6748489656-fxwlp" (UID: "03156781-088a-4815-bcfd-cdc309e27c13") : secret "samples-operator-tls" not found
Jul 24 21:16:12.505351 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.505327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6cp2\" (UniqueName: \"kubernetes.io/projected/03156781-088a-4815-bcfd-cdc309e27c13-kube-api-access-r6cp2\") pod \"cluster-samples-operator-6748489656-fxwlp\" (UID: \"03156781-088a-4815-bcfd-cdc309e27c13\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"
Jul 24 21:16:12.556451 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.556426    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj"
Jul 24 21:16:12.569565 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.569543    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ppqph"
Jul 24 21:16:12.586869 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.586847    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6"
Jul 24 21:16:12.747915 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.747844    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj"]
Jul 24 21:16:12.752398 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:16:12.752343    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf093810_8a45_4cc0_b0e2_a1839e46b254.slice/crio-8822350fad7ad02f43a3124e14fb1533f4f1a9e844bbee0d1bb308c6a63a978b WatchSource:0}: Error finding container 8822350fad7ad02f43a3124e14fb1533f4f1a9e844bbee0d1bb308c6a63a978b: Status 404 returned error can't find the container with id 8822350fad7ad02f43a3124e14fb1533f4f1a9e844bbee0d1bb308c6a63a978b
Jul 24 21:16:12.764787 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.764761    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ppqph"]
Jul 24 21:16:12.768341 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:16:12.768314    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8acf81_3fb0_4095_9ae4_b9b514ffffe1.slice/crio-f04adaacdeb3739b37ad5f6317c3ae855be11b522cc3ec2466e8984b99d7369e WatchSource:0}: Error finding container f04adaacdeb3739b37ad5f6317c3ae855be11b522cc3ec2466e8984b99d7369e: Status 404 returned error can't find the container with id f04adaacdeb3739b37ad5f6317c3ae855be11b522cc3ec2466e8984b99d7369e
Jul 24 21:16:12.784721 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.784701    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6"]
Jul 24 21:16:12.788629 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:16:12.788605    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38bf3087_9101_4b71_be34_dd0726961e2a.slice/crio-4c46bcda7568308332aeeba6e749b4c15a9e76497f9e30c650365c0268740c95 WatchSource:0}: Error finding container 4c46bcda7568308332aeeba6e749b4c15a9e76497f9e30c650365c0268740c95: Status 404 returned error can't find the container with id 4c46bcda7568308332aeeba6e749b4c15a9e76497f9e30c650365c0268740c95
Jul 24 21:16:12.999666 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:12.999577    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-fxwlp\" (UID: \"03156781-088a-4815-bcfd-cdc309e27c13\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"
Jul 24 21:16:12.999824 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:12.999720    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 21:16:12.999824 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:12.999792    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls podName:03156781-088a-4815-bcfd-cdc309e27c13 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:13.999773369 +0000 UTC m=+52.756119179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls") pod "cluster-samples-operator-6748489656-fxwlp" (UID: "03156781-088a-4815-bcfd-cdc309e27c13") : secret "samples-operator-tls" not found
Jul 24 21:16:13.093798 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:13.093761    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ppqph" event={"ID":"bb8acf81-3fb0-4095-9ae4-b9b514ffffe1","Type":"ContainerStarted","Data":"f04adaacdeb3739b37ad5f6317c3ae855be11b522cc3ec2466e8984b99d7369e"}
Jul 24 21:16:13.094843 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:13.094822    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6" event={"ID":"38bf3087-9101-4b71-be34-dd0726961e2a","Type":"ContainerStarted","Data":"4c46bcda7568308332aeeba6e749b4c15a9e76497f9e30c650365c0268740c95"}
Jul 24 21:16:13.095834 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:13.095818    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj" event={"ID":"df093810-8a45-4cc0-b0e2-a1839e46b254","Type":"ContainerStarted","Data":"8822350fad7ad02f43a3124e14fb1533f4f1a9e844bbee0d1bb308c6a63a978b"}
Jul 24 21:16:13.502898 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:13.502860    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:13.503496 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:13.502927    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:13.503496 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:13.503063    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 21:16:13.503496 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:13.503119    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:13.503496 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:13.503151    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm podName:d8999bbe-7e48-4678-8951-b232a39f1de9 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:21.503114869 +0000 UTC m=+60.259460660 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r75fx" (UID: "d8999bbe-7e48-4678-8951-b232a39f1de9") : configmap "kube-rbac-proxy" not found
Jul 24 21:16:13.503496 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:13.503197    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls podName:d8999bbe-7e48-4678-8951-b232a39f1de9 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:21.503179696 +0000 UTC m=+60.259525492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r75fx" (UID: "d8999bbe-7e48-4678-8951-b232a39f1de9") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:14.007194 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:14.007164    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-fxwlp\" (UID: \"03156781-088a-4815-bcfd-cdc309e27c13\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"
Jul 24 21:16:14.007391 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:14.007374    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 21:16:14.007519 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:14.007434    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls podName:03156781-088a-4815-bcfd-cdc309e27c13 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:16.007415759 +0000 UTC m=+54.763761552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls") pod "cluster-samples-operator-6748489656-fxwlp" (UID: "03156781-088a-4815-bcfd-cdc309e27c13") : secret "samples-operator-tls" not found
Jul 24 21:16:14.101385 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:14.101317    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ppqph" event={"ID":"bb8acf81-3fb0-4095-9ae4-b9b514ffffe1","Type":"ContainerStarted","Data":"a9d6bfe70093ef13c10d07ad7242881c8b6af10acf2d3b51e5296f3492b8f950"}
Jul 24 21:16:14.122011 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:14.121948    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ppqph" podStartSLOduration=0.963426441 podStartE2EDuration="2.121932353s" totalImagesPullingTime="1.158505912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:14.120494206 +0000 UTC m=+52.876840021" watchObservedRunningTime="2026-07-24 21:16:14.121932353 +0000 UTC m=+52.878278179"
Jul 24 21:16:14.213224 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:14.213106    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:14.292517 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:14.292481    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:15.988698 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:15.988666    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:16.023081 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:16.023053    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-fxwlp\" (UID: \"03156781-088a-4815-bcfd-cdc309e27c13\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"
Jul 24 21:16:16.023270 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:16.023211    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 21:16:16.023321 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:16.023273    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls podName:03156781-088a-4815-bcfd-cdc309e27c13 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:20.023258019 +0000 UTC m=+58.779603810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls") pod "cluster-samples-operator-6748489656-fxwlp" (UID: "03156781-088a-4815-bcfd-cdc309e27c13") : secret "samples-operator-tls" not found
Jul 24 21:16:16.091635 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:16.091597    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:16.111173 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:16.111146    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj" event={"ID":"df093810-8a45-4cc0-b0e2-a1839e46b254","Type":"ContainerStarted","Data":"b3acea82b8154310c86cddfc3b2460b7cfd5646adf40900974427731b6109ff3"}
Jul 24 21:16:16.112584 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:16.112559    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6" event={"ID":"38bf3087-9101-4b71-be34-dd0726961e2a","Type":"ContainerStarted","Data":"290fc800c282348951d3a292bd70ecc464f610eb79adf5bd8e1e3ee4fb21680a"}
Jul 24 21:16:16.138671 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:16.138528    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj" podStartSLOduration=1.850182623 podStartE2EDuration="4.138510932s" totalImagesPullingTime="2.288328309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:16.137612775 +0000 UTC m=+54.893958588" watchObservedRunningTime="2026-07-24 21:16:16.138510932 +0000 UTC m=+54.894856745"
Jul 24 21:16:16.169979 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:16.169935    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6" podStartSLOduration=1.91408888 podStartE2EDuration="4.169923822s" totalImagesPullingTime="2.255834942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:16.168350622 +0000 UTC m=+54.924696435" watchObservedRunningTime="2026-07-24 21:16:16.169923822 +0000 UTC m=+54.926269643"
Jul 24 21:16:16.176779 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:16.176755    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:16.240121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:16.240057    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:16.691557 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:16.691524    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:16.791473 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:16.791443    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:17.167019 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:17.166972    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:17.233349 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:17.233319    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:18.141300 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:18.141266    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vlf8v\" (UID: \"c944d753-b675-4664-944a-3ec3c10b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:18.141464 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:18.141406    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 24 21:16:18.141508 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:18.141471    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls podName:c944d753-b675-4664-944a-3ec3c10b224f nodeName:}" failed. No retries permitted until 2026-07-24 21:16:34.14145509 +0000 UTC m=+72.897800881 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vlf8v" (UID: "c944d753-b675-4664-944a-3ec3c10b224f") : secret "cluster-monitoring-operator-tls" not found
Jul 24 21:16:18.370997 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:18.370968    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:18.431879 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:18.431805    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:18.802492 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:18.802408    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:18.929407 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:18.929374    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:19.036061 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:19.036027    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:19.288720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:19.288688    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:20.058695 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:20.058664    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-fxwlp\" (UID: \"03156781-088a-4815-bcfd-cdc309e27c13\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"
Jul 24 21:16:20.059090 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:20.058778    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 24 21:16:20.059090 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:20.058843    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls podName:03156781-088a-4815-bcfd-cdc309e27c13 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:28.058828683 +0000 UTC m=+66.815174474 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls") pod "cluster-samples-operator-6748489656-fxwlp" (UID: "03156781-088a-4815-bcfd-cdc309e27c13") : secret "samples-operator-tls" not found
Jul 24 21:16:21.433336 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:21.433302    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:21.571067 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:21.571028    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:21.571275 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:21.571092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:21.571275 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:21.571208    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:21.571433 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:21.571280    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls podName:d8999bbe-7e48-4678-8951-b232a39f1de9 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:37.571264633 +0000 UTC m=+76.327610425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r75fx" (UID: "d8999bbe-7e48-4678-8951-b232a39f1de9") : secret "insights-runtime-extractor-tls" not found
Jul 24 21:16:21.571529 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:21.571511    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d8999bbe-7e48-4678-8951-b232a39f1de9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:21.607086 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:21.607049    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:21.706103 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:21.706034    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:21.815274 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:21.815240    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:21.917879 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:21.917834    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:22.018494 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:22.018417    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:22.128714 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:22.128671    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:22.341913 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:22.341825    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:22.724994 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:22.724961    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:23.553571 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:23.553534    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:23.767693 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:23.767660    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:24.047239 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:24.047216    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-88v79"
Jul 24 21:16:24.139908 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:24.139883    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:24.348959 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:24.348879    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:24.732845 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:24.732808    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:24.835107 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:24.835078    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:24.938549 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:24.938527    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:25.266774 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:25.266744    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:25.521917 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:25.521849    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:25.933678 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:25.933640    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:26.123877 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:26.123850    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:26.224874 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:26.224796    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:26.325109 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:26.325068    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:26.526359 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:26.526276    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:27.421568 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.421541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:16:27.424199 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.424182    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 21:16:27.434822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.434791    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1ec4ede0-f187-4d6c-a8a2-dd853f431b01-original-pull-secret\") pod \"global-pull-secret-syncer-b868q\" (UID: \"1ec4ede0-f187-4d6c-a8a2-dd853f431b01\") " pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:16:27.464905 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.464883    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b868q"
Jul 24 21:16:27.520962 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.520936    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:27.521994 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.521930    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs\") pod \"network-metrics-daemon-ldqjs\" (UID: \"de44a92d-8440-4b1f-a0ba-479919425485\") " pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:16:27.524588 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.524558    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 21:16:27.532948 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:27.532928    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 24 21:16:27.533040 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:27.532998    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs podName:de44a92d-8440-4b1f-a0ba-479919425485 nodeName:}" failed. No retries permitted until 2026-07-24 21:17:31.532975647 +0000 UTC m=+130.289321452 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs") pod "network-metrics-daemon-ldqjs" (UID: "de44a92d-8440-4b1f-a0ba-479919425485") : secret "metrics-daemon-secret" not found
Jul 24 21:16:27.589596 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.589565    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b868q"]
Jul 24 21:16:27.592849 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:16:27.592812    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec4ede0_f187_4d6c_a8a2_dd853f431b01.slice/crio-c1d703859058f0a243a951179fea43f2f47f6fc737e70b60760dbe42fb079d24 WatchSource:0}: Error finding container c1d703859058f0a243a951179fea43f2f47f6fc737e70b60760dbe42fb079d24: Status 404 returned error can't find the container with id c1d703859058f0a243a951179fea43f2f47f6fc737e70b60760dbe42fb079d24
Jul 24 21:16:27.623015 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.622992    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9vcq\" (UniqueName: \"kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq\") pod \"network-check-target-cpmnw\" (UID: \"0f1880b1-1b8e-4633-841a-ed19c927a53d\") " pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:16:27.626224 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.626205    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 21:16:27.635664 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.635642    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 21:16:27.646246 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.646218    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9vcq\" (UniqueName: \"kubernetes.io/projected/0f1880b1-1b8e-4633-841a-ed19c927a53d-kube-api-access-v9vcq\") pod \"network-check-target-cpmnw\" (UID: \"0f1880b1-1b8e-4633-841a-ed19c927a53d\") " pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:16:27.773385 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.773299    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n9q4d\""
Jul 24 21:16:27.781589 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.781568    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:16:27.909956 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.909925    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cpmnw"]
Jul 24 21:16:27.913694 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:16:27.913670    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1880b1_1b8e_4633_841a_ed19c927a53d.slice/crio-1ef98096cea68a59eebdb990169dc3745b5b41c1436bea59799cd4016ca0a376 WatchSource:0}: Error finding container 1ef98096cea68a59eebdb990169dc3745b5b41c1436bea59799cd4016ca0a376: Status 404 returned error can't find the container with id 1ef98096cea68a59eebdb990169dc3745b5b41c1436bea59799cd4016ca0a376
Jul 24 21:16:27.916350 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:27.916328    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:28.119551 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.119510    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:28.127274 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.127243    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-fxwlp\" (UID: \"03156781-088a-4815-bcfd-cdc309e27c13\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"
Jul 24 21:16:28.130052 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.130028    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/03156781-088a-4815-bcfd-cdc309e27c13-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-fxwlp\" (UID: \"03156781-088a-4815-bcfd-cdc309e27c13\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"
Jul 24 21:16:28.143429 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.143395    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b868q" event={"ID":"1ec4ede0-f187-4d6c-a8a2-dd853f431b01","Type":"ContainerStarted","Data":"c1d703859058f0a243a951179fea43f2f47f6fc737e70b60760dbe42fb079d24"}
Jul 24 21:16:28.144672 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.144647    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cpmnw" event={"ID":"0f1880b1-1b8e-4633-841a-ed19c927a53d","Type":"ContainerStarted","Data":"1ef98096cea68a59eebdb990169dc3745b5b41c1436bea59799cd4016ca0a376"}
Jul 24 21:16:28.214605 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.214578    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6qmgl\""
Jul 24 21:16:28.222812 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.222792    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"
Jul 24 21:16:28.243830 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.243805    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:28.346614 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.346527    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:28.380834 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.380620    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp"]
Jul 24 21:16:28.520205 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.520097    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:28.530308 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.530277    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:16:28.533222 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.533197    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b0ade7d3-1225-4c63-8282-4b0ea5537005-metrics-tls\") pod \"dns-default-j8cqc\" (UID: \"b0ade7d3-1225-4c63-8282-4b0ea5537005\") " pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:16:28.725026 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.724941    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-thbdw\""
Jul 24 21:16:28.732331 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.732307    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:16:28.887150 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:28.887105    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j8cqc"]
Jul 24 21:16:28.894415 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:16:28.894385    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ade7d3_1225_4c63_8282_4b0ea5537005.slice/crio-907fff6fe9038fd03c64d0519125a118cea897815d1d3351c14f6816b3cf29b2 WatchSource:0}: Error finding container 907fff6fe9038fd03c64d0519125a118cea897815d1d3351c14f6816b3cf29b2: Status 404 returned error can't find the container with id 907fff6fe9038fd03c64d0519125a118cea897815d1d3351c14f6816b3cf29b2
Jul 24 21:16:29.150041 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:29.150003    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j8cqc" event={"ID":"b0ade7d3-1225-4c63-8282-4b0ea5537005","Type":"ContainerStarted","Data":"907fff6fe9038fd03c64d0519125a118cea897815d1d3351c14f6816b3cf29b2"}
Jul 24 21:16:29.151604 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:29.151553    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp" event={"ID":"03156781-088a-4815-bcfd-cdc309e27c13","Type":"ContainerStarted","Data":"b79287f4b03e7841c9e8f2d6b595eaa40a36b1fe0326d9684e9873606978d6a6"}
Jul 24 21:16:29.727251 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:29.727149    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:29.925169 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:29.925115    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:30.118861 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:30.118819    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:30.322373 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:30.322335    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:30.530600 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:30.530515    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:30.859286 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:30.859246    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:31.526264 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:31.526216    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:31.718584 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:31.718539    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:31.925311 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:31.925264    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:32.131908 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:32.131873    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:34.173246 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.173208    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b868q" event={"ID":"1ec4ede0-f187-4d6c-a8a2-dd853f431b01","Type":"ContainerStarted","Data":"dea28bc02ea00c2063d77198a2c4d47c9c487f3fcb3d292924be92684caefd25"}
Jul 24 21:16:34.174845 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.174824    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j8cqc" event={"ID":"b0ade7d3-1225-4c63-8282-4b0ea5537005","Type":"ContainerStarted","Data":"c3ac92e86ca496dfd08aa40a62058b0defe984fdcf12a5c4b8c3352d46cd420f"}
Jul 24 21:16:34.174945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.174868    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j8cqc" event={"ID":"b0ade7d3-1225-4c63-8282-4b0ea5537005","Type":"ContainerStarted","Data":"9d0d2138a9487139f7b9e3c0c73ba1de1c55dd024e490c8c83c4cbf7a9d475ad"}
Jul 24 21:16:34.174945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.174900    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:16:34.176623 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.176601    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp" event={"ID":"03156781-088a-4815-bcfd-cdc309e27c13","Type":"ContainerStarted","Data":"fda47b3688d03cdbd8d822f0f9565372cedd83de4ddcf1f624da77cb94b3d671"}
Jul 24 21:16:34.176734 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.176643    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp" event={"ID":"03156781-088a-4815-bcfd-cdc309e27c13","Type":"ContainerStarted","Data":"14df16c356f6915dc7b57aaa17dcab5a4e130e365d94279c63c51021dcc340d8"}
Jul 24 21:16:34.178363 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.178334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cpmnw" event={"ID":"0f1880b1-1b8e-4633-841a-ed19c927a53d","Type":"ContainerStarted","Data":"94f4ca0ab2214fd2ac6d44f9485600559c1fe09cb583368844ec5d225f1532d9"}
Jul 24 21:16:34.178455 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.178441    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:16:34.183045 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.183028    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vlf8v\" (UID: \"c944d753-b675-4664-944a-3ec3c10b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:34.185471 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.185448    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c944d753-b675-4664-944a-3ec3c10b224f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vlf8v\" (UID: \"c944d753-b675-4664-944a-3ec3c10b224f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:34.195955 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.195922    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b868q" podStartSLOduration=66.520699209 podStartE2EDuration="1m12.195909736s" totalImagesPullingTime="5.675210527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:34.195103914 +0000 UTC m=+72.951449726" watchObservedRunningTime="2026-07-24 21:16:34.195909736 +0000 UTC m=+72.952255549"
Jul 24 21:16:34.235624 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.235580    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cpmnw" podStartSLOduration=66.889324483 podStartE2EDuration="1m12.235568362s" totalImagesPullingTime="5.346243879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:34.23402984 +0000 UTC m=+72.990375652" watchObservedRunningTime="2026-07-24 21:16:34.235568362 +0000 UTC m=+72.991914235"
Jul 24 21:16:34.235740 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.235705    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-j8cqc" podStartSLOduration=33.874607095 podStartE2EDuration="38.235696111s" totalImagesPullingTime="4.361089016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:34.216081838 +0000 UTC m=+72.972427644" watchObservedRunningTime="2026-07-24 21:16:34.235696111 +0000 UTC m=+72.992041926"
Jul 24 21:16:34.354745 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.354713    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-mrlmd\""
Jul 24 21:16:34.361644 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.361622    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"
Jul 24 21:16:34.491332 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.491274    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-fxwlp" podStartSLOduration=17.66458298 podStartE2EDuration="22.491249863s" totalImagesPullingTime="4.826666883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:34.254695587 +0000 UTC m=+73.011041400" watchObservedRunningTime="2026-07-24 21:16:34.491249863 +0000 UTC m=+73.247595677"
Jul 24 21:16:34.491579 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:34.491562    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v"]
Jul 24 21:16:34.494461 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:16:34.494436    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc944d753_b675_4664_944a_3ec3c10b224f.slice/crio-d93044af9284dcda914268a4cd2fdc31e2eeff72604c2ac04fa1606e1527a4e4 WatchSource:0}: Error finding container d93044af9284dcda914268a4cd2fdc31e2eeff72604c2ac04fa1606e1527a4e4: Status 404 returned error can't find the container with id d93044af9284dcda914268a4cd2fdc31e2eeff72604c2ac04fa1606e1527a4e4
Jul 24 21:16:35.185098 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:35.184663    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v" event={"ID":"c944d753-b675-4664-944a-3ec3c10b224f","Type":"ContainerStarted","Data":"d93044af9284dcda914268a4cd2fdc31e2eeff72604c2ac04fa1606e1527a4e4"}
Jul 24 21:16:36.475930 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:36.475845    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:36.651638 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:36.651607    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:36.811146 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:36.811049    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:37.192265 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:37.192235    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v" event={"ID":"c944d753-b675-4664-944a-3ec3c10b224f","Type":"ContainerStarted","Data":"e171f7f126a5ed4bbc9f2efceba30de464326d2987c91ce429da29c66b3826c6"}
Jul 24 21:16:37.211283 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:37.211240    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vlf8v" podStartSLOduration=33.550262909 podStartE2EDuration="35.211224895s" totalImagesPullingTime="1.660961986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:37.210643569 +0000 UTC m=+75.966989393" watchObservedRunningTime="2026-07-24 21:16:37.211224895 +0000 UTC m=+75.967570735"
Jul 24 21:16:37.608920 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:37.608888    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:37.611506 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:37.611478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d8999bbe-7e48-4678-8951-b232a39f1de9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r75fx\" (UID: \"d8999bbe-7e48-4678-8951-b232a39f1de9\") " pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:37.840988 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:37.840960    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4d5n4\""
Jul 24 21:16:37.849026 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:37.849004    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r75fx"
Jul 24 21:16:37.979791 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:37.979763    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r75fx"]
Jul 24 21:16:37.983097 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:16:37.983071    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8999bbe_7e48_4678_8951_b232a39f1de9.slice/crio-7c2265c532ce8985c73f5c50986f2d44d1ebd67ecb6c12b7733f87bad56a8b31 WatchSource:0}: Error finding container 7c2265c532ce8985c73f5c50986f2d44d1ebd67ecb6c12b7733f87bad56a8b31: Status 404 returned error can't find the container with id 7c2265c532ce8985c73f5c50986f2d44d1ebd67ecb6c12b7733f87bad56a8b31
Jul 24 21:16:38.197732 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:38.197650    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r75fx" event={"ID":"d8999bbe-7e48-4678-8951-b232a39f1de9","Type":"ContainerStarted","Data":"7c2265c532ce8985c73f5c50986f2d44d1ebd67ecb6c12b7733f87bad56a8b31"}
Jul 24 21:16:39.829922 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:39.829884    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:39.890250 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:39.890169    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:40.356587 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.356547    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"]
Jul 24 21:16:40.368179 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.368153    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.370811 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.370785    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 24 21:16:40.370963 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.370937    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 24 21:16:40.371144 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.371103    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 24 21:16:40.371805 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.371787    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ms492\""
Jul 24 21:16:40.373938 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.373915    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"]
Jul 24 21:16:40.378049 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.377972    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 24 21:16:40.394517 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.394490    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"]
Jul 24 21:16:40.394717 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:40.394697    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-xmlvq registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m" podUID="5e52f352-d8f4-4fdc-9c9a-678f425e74a6"
Jul 24 21:16:40.431363 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.431329    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-registry-tls\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.431508 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.431398    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-bound-sa-token\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.431508 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.431488    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-trusted-ca\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.431597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.431540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmlvq\" (UniqueName: \"kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-kube-api-access-xmlvq\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.431635 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.431610    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-image-registry-private-configuration\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.431683 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.431639    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-ca-trust-extracted\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.431683 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.431668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-registry-certificates\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.431773 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.431687    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-installation-pull-secrets\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.474864 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.474829    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-926h2"]
Jul 24 21:16:40.479856 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.479834    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-926h2"
Jul 24 21:16:40.484575 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.484550    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nmslf\""
Jul 24 21:16:40.484880 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.484856    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 24 21:16:40.494823 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.494800    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:40.508871 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.508833    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-926h2"]
Jul 24 21:16:40.532214 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.532184    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-bound-sa-token\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.532334 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.532227    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-trusted-ca\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.532334 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.532271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmlvq\" (UniqueName: \"kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-kube-api-access-xmlvq\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.532334 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.532307    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-image-registry-private-configuration\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.532496 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.532358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-ca-trust-extracted\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.532496 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.532419    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-registry-certificates\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.532496 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.532451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-installation-pull-secrets\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.532496 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.532483    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-registry-tls\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.533203 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.533180    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-ca-trust-extracted\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.533315 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.533287    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-registry-certificates\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.533865 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.533842    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-trusted-ca\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.535120 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.535092    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-image-registry-private-configuration\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.535708 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.535689    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-registry-tls\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.536926 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.536906    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-installation-pull-secrets\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.577059 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.577035    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-bound-sa-token\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.583278 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.583258    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmlvq\" (UniqueName: \"kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-kube-api-access-xmlvq\") pod \"image-registry-d9cdbdd8f-gsm9m\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") " pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:40.629797 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.629776    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:40.633569 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.633542    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9ff33b94-17d1-4fd2-a256-6479f169b8b7-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-926h2\" (UID: \"9ff33b94-17d1-4fd2-a256-6479f169b8b7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-926h2"
Jul 24 21:16:40.734608 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.734575    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9ff33b94-17d1-4fd2-a256-6479f169b8b7-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-926h2\" (UID: \"9ff33b94-17d1-4fd2-a256-6479f169b8b7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-926h2"
Jul 24 21:16:40.737526 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.737493    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9ff33b94-17d1-4fd2-a256-6479f169b8b7-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-926h2\" (UID: \"9ff33b94-17d1-4fd2-a256-6479f169b8b7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-926h2"
Jul 24 21:16:40.791752 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:40.791717    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-926h2"
Jul 24 21:16:41.212141 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.211590    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:41.225455 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.225432    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:41.337819 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.337785    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-registry-certificates\") pod \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") "
Jul 24 21:16:41.337991 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.337838    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-ca-trust-extracted\") pod \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") "
Jul 24 21:16:41.337991 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.337870    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-registry-tls\") pod \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") "
Jul 24 21:16:41.337991 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.337913    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-trusted-ca\") pod \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") "
Jul 24 21:16:41.337991 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.337943    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-bound-sa-token\") pod \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") "
Jul 24 21:16:41.337991 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.337977    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-installation-pull-secrets\") pod \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") "
Jul 24 21:16:41.338257 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.338005    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmlvq\" (UniqueName: \"kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-kube-api-access-xmlvq\") pod \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") "
Jul 24 21:16:41.338257 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.338033    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-image-registry-private-configuration\") pod \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\" (UID: \"5e52f352-d8f4-4fdc-9c9a-678f425e74a6\") "
Jul 24 21:16:41.338257 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.338101    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5e52f352-d8f4-4fdc-9c9a-678f425e74a6" (UID: "5e52f352-d8f4-4fdc-9c9a-678f425e74a6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:16:41.338462 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.338275    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-ca-trust-extracted\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:16:41.338462 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.338275    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5e52f352-d8f4-4fdc-9c9a-678f425e74a6" (UID: "5e52f352-d8f4-4fdc-9c9a-678f425e74a6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:16:41.338462 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.338374    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5e52f352-d8f4-4fdc-9c9a-678f425e74a6" (UID: "5e52f352-d8f4-4fdc-9c9a-678f425e74a6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:16:41.340740 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.340713    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5e52f352-d8f4-4fdc-9c9a-678f425e74a6" (UID: "5e52f352-d8f4-4fdc-9c9a-678f425e74a6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:16:41.340852 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.340775    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5e52f352-d8f4-4fdc-9c9a-678f425e74a6" (UID: "5e52f352-d8f4-4fdc-9c9a-678f425e74a6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:16:41.340852 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.340813    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5e52f352-d8f4-4fdc-9c9a-678f425e74a6" (UID: "5e52f352-d8f4-4fdc-9c9a-678f425e74a6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:16:41.340948 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.340861    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-kube-api-access-xmlvq" (OuterVolumeSpecName: "kube-api-access-xmlvq") pod "5e52f352-d8f4-4fdc-9c9a-678f425e74a6" (UID: "5e52f352-d8f4-4fdc-9c9a-678f425e74a6"). InnerVolumeSpecName "kube-api-access-xmlvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:16:41.341014 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.340994    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5e52f352-d8f4-4fdc-9c9a-678f425e74a6" (UID: "5e52f352-d8f4-4fdc-9c9a-678f425e74a6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:16:41.438788 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.438754    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-trusted-ca\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:16:41.438788 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.438780    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-bound-sa-token\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:16:41.438788 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.438790    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-installation-pull-secrets\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:16:41.439055 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.438804    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmlvq\" (UniqueName: \"kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-kube-api-access-xmlvq\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:16:41.439055 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.438821    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-image-registry-private-configuration\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:16:41.439055 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.438837    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-registry-certificates\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:16:41.439055 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:41.438850    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e52f352-d8f4-4fdc-9c9a-678f425e74a6-registry-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:16:42.215464 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:42.215432    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"
Jul 24 21:16:42.250158 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:42.250113    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"]
Jul 24 21:16:42.254668 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:42.254644    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d9cdbdd8f-gsm9m"]
Jul 24 21:16:42.861044 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:42.861022    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-926h2"]
Jul 24 21:16:42.864490 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:16:42.864465    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff33b94_17d1_4fd2_a256_6479f169b8b7.slice/crio-48c8ce334fc0e37f79427e3f2cd5859a1bf2ab3fc831f436d9497813156fb84b WatchSource:0}: Error finding container 48c8ce334fc0e37f79427e3f2cd5859a1bf2ab3fc831f436d9497813156fb84b: Status 404 returned error can't find the container with id 48c8ce334fc0e37f79427e3f2cd5859a1bf2ab3fc831f436d9497813156fb84b
Jul 24 21:16:43.219813 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:43.219776    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r75fx" event={"ID":"d8999bbe-7e48-4678-8951-b232a39f1de9","Type":"ContainerStarted","Data":"43c3e8be257929891fddafed9777e95497ea74b1e6599e594ae7f5c2fc0216e7"}
Jul 24 21:16:43.220852 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:43.220831    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-926h2" event={"ID":"9ff33b94-17d1-4fd2-a256-6479f169b8b7","Type":"ContainerStarted","Data":"48c8ce334fc0e37f79427e3f2cd5859a1bf2ab3fc831f436d9497813156fb84b"}
Jul 24 21:16:43.755575 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:43.755538    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e52f352-d8f4-4fdc-9c9a-678f425e74a6" path="/var/lib/kubelet/pods/5e52f352-d8f4-4fdc-9c9a-678f425e74a6/volumes"
Jul 24 21:16:44.187912 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.187440    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j8cqc"
Jul 24 21:16:44.228916 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.228884    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-926h2" event={"ID":"9ff33b94-17d1-4fd2-a256-6479f169b8b7","Type":"ContainerStarted","Data":"6e3540e784869f220dbde0da0eab2866068d58c600b205d7672a3ac517a96820"}
Jul 24 21:16:44.229361 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.229080    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-926h2"
Jul 24 21:16:44.234137 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.234103    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-926h2"
Jul 24 21:16:44.253804 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.253756    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-926h2" podStartSLOduration=3.016349948 podStartE2EDuration="4.253736603s" totalImagesPullingTime="1.237386655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:44.253346529 +0000 UTC m=+83.009692356" watchObservedRunningTime="2026-07-24 21:16:44.253736603 +0000 UTC m=+83.010082417"
Jul 24 21:16:44.685879 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.685840    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:44.738676 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.738631    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-2glc5"]
Jul 24 21:16:44.749159 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.749110    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5"
Jul 24 21:16:44.751321 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.751285    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-2glc5"]
Jul 24 21:16:44.752586 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.752565    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 24 21:16:44.752720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.752587    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 21:16:44.752720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.752632    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2bch5\""
Jul 24 21:16:44.752720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.752646    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 24 21:16:44.761917 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.761898    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:44.865227 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.865194    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9330697e-c2ed-4687-a8db-8c99ed55b0a8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-2glc5\" (UID: \"9330697e-c2ed-4687-a8db-8c99ed55b0a8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5"
Jul 24 21:16:44.865381 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.865246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9330697e-c2ed-4687-a8db-8c99ed55b0a8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-2glc5\" (UID: \"9330697e-c2ed-4687-a8db-8c99ed55b0a8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5"
Jul 24 21:16:44.865381 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.865310    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq9cq\" (UniqueName: \"kubernetes.io/projected/9330697e-c2ed-4687-a8db-8c99ed55b0a8-kube-api-access-sq9cq\") pod \"prometheus-operator-84d69f794d-2glc5\" (UID: \"9330697e-c2ed-4687-a8db-8c99ed55b0a8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5"
Jul 24 21:16:44.865381 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.865367    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9330697e-c2ed-4687-a8db-8c99ed55b0a8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-2glc5\" (UID: \"9330697e-c2ed-4687-a8db-8c99ed55b0a8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5"
Jul 24 21:16:44.966747 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.966651    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9330697e-c2ed-4687-a8db-8c99ed55b0a8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-2glc5\" (UID: \"9330697e-c2ed-4687-a8db-8c99ed55b0a8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5"
Jul 24 21:16:44.966747 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.966704    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9330697e-c2ed-4687-a8db-8c99ed55b0a8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-2glc5\" (UID: \"9330697e-c2ed-4687-a8db-8c99ed55b0a8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5"
Jul 24 21:16:44.966747 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.966740    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq9cq\" (UniqueName: \"kubernetes.io/projected/9330697e-c2ed-4687-a8db-8c99ed55b0a8-kube-api-access-sq9cq\") pod \"prometheus-operator-84d69f794d-2glc5\" (UID: \"9330697e-c2ed-4687-a8db-8c99ed55b0a8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5"
Jul 24 21:16:44.967036 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.966766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9330697e-c2ed-4687-a8db-8c99ed55b0a8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-2glc5\" (UID: \"9330697e-c2ed-4687-a8db-8c99ed55b0a8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5"
Jul 24 21:16:44.967036 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:44.966820    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 24 21:16:44.967036 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:44.966912    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9330697e-c2ed-4687-a8db-8c99ed55b0a8-prometheus-operator-tls podName:9330697e-c2ed-4687-a8db-8c99ed55b0a8 nodeName:}" failed. No retries permitted until 2026-07-24 21:16:45.46689672 +0000 UTC m=+84.223242514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/9330697e-c2ed-4687-a8db-8c99ed55b0a8-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-2glc5" (UID: "9330697e-c2ed-4687-a8db-8c99ed55b0a8") : secret "prometheus-operator-tls" not found
Jul 24 21:16:44.967525 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.967498    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9330697e-c2ed-4687-a8db-8c99ed55b0a8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-2glc5\" (UID: \"9330697e-c2ed-4687-a8db-8c99ed55b0a8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5"
Jul 24 21:16:44.969572 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.969542    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9330697e-c2ed-4687-a8db-8c99ed55b0a8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-2glc5\" (UID: \"9330697e-c2ed-4687-a8db-8c99ed55b0a8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5"
Jul 24 21:16:44.976359 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:44.976335    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq9cq\" (UniqueName: \"kubernetes.io/projected/9330697e-c2ed-4687-a8db-8c99ed55b0a8-kube-api-access-sq9cq\") pod \"prometheus-operator-84d69f794d-2glc5\" (UID: \"9330697e-c2ed-4687-a8db-8c99ed55b0a8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5"
Jul 24 21:16:45.461726 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:45.461684    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:45.469740 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:45.469702    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9330697e-c2ed-4687-a8db-8c99ed55b0a8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-2glc5\" (UID: \"9330697e-c2ed-4687-a8db-8c99ed55b0a8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5"
Jul 24 21:16:45.472515 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:45.472490    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9330697e-c2ed-4687-a8db-8c99ed55b0a8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-2glc5\" (UID: \"9330697e-c2ed-4687-a8db-8c99ed55b0a8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5"
Jul 24 21:16:45.661736 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:45.661701    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5"
Jul 24 21:16:46.023087 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:46.023053    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:46.088552 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:46.088522    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:46.397959 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:46.397857    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-2glc5"]
Jul 24 21:16:46.400241 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:16:46.400216    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9330697e_c2ed_4687_a8db_8c99ed55b0a8.slice/crio-34d18cd66cdf9a25931af7c2b2674a91eeeb994976fc2853b15a4f45c8833ab4 WatchSource:0}: Error finding container 34d18cd66cdf9a25931af7c2b2674a91eeeb994976fc2853b15a4f45c8833ab4: Status 404 returned error can't find the container with id 34d18cd66cdf9a25931af7c2b2674a91eeeb994976fc2853b15a4f45c8833ab4
Jul 24 21:16:47.241066 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:47.241030    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5" event={"ID":"9330697e-c2ed-4687-a8db-8c99ed55b0a8","Type":"ContainerStarted","Data":"34d18cd66cdf9a25931af7c2b2674a91eeeb994976fc2853b15a4f45c8833ab4"}
Jul 24 21:16:47.243317 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:47.243289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r75fx" event={"ID":"d8999bbe-7e48-4678-8951-b232a39f1de9","Type":"ContainerStarted","Data":"9b4144cc27081be5b218e0416ddf1844a0d33d448038b18f85c6758746413caf"}
Jul 24 21:16:48.107864 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:48.107839    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:48.249559 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:48.249517    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5" event={"ID":"9330697e-c2ed-4687-a8db-8c99ed55b0a8","Type":"ContainerStarted","Data":"9407020420dccd2e50649a257154acd191f89f6cd8e5ddd4436d517798159d82"}
Jul 24 21:16:48.249993 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:48.249588    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5" event={"ID":"9330697e-c2ed-4687-a8db-8c99ed55b0a8","Type":"ContainerStarted","Data":"c03cccf49ac4adb772e9c1b53c000d8c97c0abfe9813e106b9d2852636cc4556"}
Jul 24 21:16:48.269914 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:48.269832    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-2glc5" podStartSLOduration=2.852024861 podStartE2EDuration="4.269810874s" totalImagesPullingTime="1.417786013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:48.268542421 +0000 UTC m=+87.024888299" watchObservedRunningTime="2026-07-24 21:16:48.269810874 +0000 UTC m=+87.026156689"
Jul 24 21:16:49.861753 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:49.861722    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:49.999903 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:49.999874    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:50.108288 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.108258    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:50.131058 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.131037    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"]
Jul 24 21:16:50.134025 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.134009    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.135109 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.135090    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qq55j"]
Jul 24 21:16:50.136609 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.136590    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lcznx\""
Jul 24 21:16:50.136729 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.136712    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 24 21:16:50.138103 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.138087    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.138956 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.138934    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 24 21:16:50.139119 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.139106    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 24 21:16:50.142950 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.142933    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 21:16:50.143034 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.142963    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 21:16:50.143374 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.143358    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 21:16:50.143615 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.143601    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ngxl2\""
Jul 24 21:16:50.151729 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.151709    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"]
Jul 24 21:16:50.194817 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.194787    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:50.212360 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.212330    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svb4x\" (UniqueName: \"kubernetes.io/projected/dd2f032e-b417-4b44-8208-c885789525df-kube-api-access-svb4x\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.212360 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.212362    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd2f032e-b417-4b44-8208-c885789525df-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.212542 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.212383    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dd2f032e-b417-4b44-8208-c885789525df-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.212542 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.212410    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd2f032e-b417-4b44-8208-c885789525df-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.212542 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.212472    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dd2f032e-b417-4b44-8208-c885789525df-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.212542 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.212510    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd2f032e-b417-4b44-8208-c885789525df-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.258586 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.258546    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r75fx" event={"ID":"d8999bbe-7e48-4678-8951-b232a39f1de9","Type":"ContainerStarted","Data":"2e378816bf59b2744d12bd57e56e380568e3c037022d94e20d6e48d8de59c571"}
Jul 24 21:16:50.277309 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.277260    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r75fx" podStartSLOduration=33.332508835 podStartE2EDuration="45.277242279s" totalImagesPullingTime="11.944733444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:16:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:50.276579916 +0000 UTC m=+89.032925730" watchObservedRunningTime="2026-07-24 21:16:50.277242279 +0000 UTC m=+89.033588092"
Jul 24 21:16:50.313852 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.313829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-metrics-client-ca\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.313979 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.313881    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svb4x\" (UniqueName: \"kubernetes.io/projected/dd2f032e-b417-4b44-8208-c885789525df-kube-api-access-svb4x\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.313979 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.313910    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-node-exporter-textfile\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.313979 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.313938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-root\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.313979 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.313966    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd2f032e-b417-4b44-8208-c885789525df-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.314181 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.314163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjncj\" (UniqueName: \"kubernetes.io/projected/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-kube-api-access-cjncj\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.314239 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.314209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dd2f032e-b417-4b44-8208-c885789525df-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.314239 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.314230    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-node-exporter-tls\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.314340 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.314266    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd2f032e-b417-4b44-8208-c885789525df-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.314340 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.314287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-node-exporter-wtmp\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.314340 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.314315    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dd2f032e-b417-4b44-8208-c885789525df-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.314498 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.314341    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd2f032e-b417-4b44-8208-c885789525df-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.314498 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.314373    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-sys\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.314498 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:50.314395    2560 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 24 21:16:50.314498 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.314401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.314498 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.314426    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-node-exporter-accelerators-collector-config\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.314498 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:16:50.314456    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd2f032e-b417-4b44-8208-c885789525df-kube-state-metrics-tls podName:dd2f032e-b417-4b44-8208-c885789525df nodeName:}" failed. No retries permitted until 2026-07-24 21:16:50.814435952 +0000 UTC m=+89.570781756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/dd2f032e-b417-4b44-8208-c885789525df-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-mq6zw" (UID: "dd2f032e-b417-4b44-8208-c885789525df") : secret "kube-state-metrics-tls" not found
Jul 24 21:16:50.314799 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.314599    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dd2f032e-b417-4b44-8208-c885789525df-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.314799 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.314681    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd2f032e-b417-4b44-8208-c885789525df-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.314868 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.314855    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dd2f032e-b417-4b44-8208-c885789525df-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.316758 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.316741    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd2f032e-b417-4b44-8208-c885789525df-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.325469 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.325447    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svb4x\" (UniqueName: \"kubernetes.io/projected/dd2f032e-b417-4b44-8208-c885789525df-kube-api-access-svb4x\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.376942 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.376848    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:50.415002 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.414971    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-metrics-client-ca\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.415193 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.415021    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-node-exporter-textfile\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.415193 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.415043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-root\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.415193 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.415084    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjncj\" (UniqueName: \"kubernetes.io/projected/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-kube-api-access-cjncj\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.415193 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.415115    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-node-exporter-tls\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.415420 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.415195    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-node-exporter-wtmp\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.415420 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.415271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-sys\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.415420 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.415299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.415420 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.415323    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-node-exporter-accelerators-collector-config\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.415620 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.415477    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-node-exporter-textfile\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.415620 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.415193    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-root\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.415620 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.415557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-sys\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.415620 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.415608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-node-exporter-wtmp\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.415792 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.415697    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-metrics-client-ca\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.415840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.415806    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-node-exporter-accelerators-collector-config\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.417709 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.417686    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-node-exporter-tls\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.417863 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.417844    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.423404 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.423383    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjncj\" (UniqueName: \"kubernetes.io/projected/14bf2715-d0e7-48ed-a231-2d6a1737f9bb-kube-api-access-cjncj\") pod \"node-exporter-qq55j\" (UID: \"14bf2715-d0e7-48ed-a231-2d6a1737f9bb\") " pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.448832 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.448810    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qq55j"
Jul 24 21:16:50.460462 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:16:50.460434    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14bf2715_d0e7_48ed_a231_2d6a1737f9bb.slice/crio-1a6cc37fd82423856bfa3e10ace5ab5302696dac83b844bedc192a68ef528417 WatchSource:0}: Error finding container 1a6cc37fd82423856bfa3e10ace5ab5302696dac83b844bedc192a68ef528417: Status 404 returned error can't find the container with id 1a6cc37fd82423856bfa3e10ace5ab5302696dac83b844bedc192a68ef528417
Jul 24 21:16:50.478140 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.478109    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:50.578656 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.578630    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:50.818241 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.818159    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd2f032e-b417-4b44-8208-c885789525df-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.820571 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.820554    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dd2f032e-b417-4b44-8208-c885789525df-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-mq6zw\" (UID: \"dd2f032e-b417-4b44-8208-c885789525df\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:50.835883 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:50.835857    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:51.043188 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.043157    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"
Jul 24 21:16:51.106824 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.106733    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:51.204972 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.204368    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw"]
Jul 24 21:16:51.206936 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.206910    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:51.209987 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:16:51.209962    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2f032e_b417_4b44_8208_c885789525df.slice/crio-ad5836e13418dd6276680cff91632119f62201dd74299f960e0d6566350c997f WatchSource:0}: Error finding container ad5836e13418dd6276680cff91632119f62201dd74299f960e0d6566350c997f: Status 404 returned error can't find the container with id ad5836e13418dd6276680cff91632119f62201dd74299f960e0d6566350c997f
Jul 24 21:16:51.238552 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.238530    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:16:51.242504 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.242487    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.246210 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.245847    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 24 21:16:51.246210 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.245862    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 24 21:16:51.246210 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.246017    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 24 21:16:51.246410 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.246264    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 24 21:16:51.247028 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.246802    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 24 21:16:51.247028 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.246848    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2fgzg\""
Jul 24 21:16:51.247236 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.247082    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 24 21:16:51.248366 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.247391    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 24 21:16:51.252213 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.252194    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 24 21:16:51.252892 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.252863    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 24 21:16:51.264291 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.264262    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw" event={"ID":"dd2f032e-b417-4b44-8208-c885789525df","Type":"ContainerStarted","Data":"ad5836e13418dd6276680cff91632119f62201dd74299f960e0d6566350c997f"}
Jul 24 21:16:51.265642 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.265620    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq55j" event={"ID":"14bf2715-d0e7-48ed-a231-2d6a1737f9bb","Type":"ContainerStarted","Data":"1a6cc37fd82423856bfa3e10ace5ab5302696dac83b844bedc192a68ef528417"}
Jul 24 21:16:51.277615 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.277591    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:16:51.289663 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.289636    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:51.322435 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.322406    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7179b107-90f0-47da-991a-3f42a83d71ac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.322567 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.322458    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.322567 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.322512    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-config-volume\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.322567 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.322536    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.322700 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.322566    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dldc4\" (UniqueName: \"kubernetes.io/projected/7179b107-90f0-47da-991a-3f42a83d71ac-kube-api-access-dldc4\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.322700 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.322613    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-web-config\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.322700 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.322639    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7179b107-90f0-47da-991a-3f42a83d71ac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.322700 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.322665    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7179b107-90f0-47da-991a-3f42a83d71ac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.322700 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.322681    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7179b107-90f0-47da-991a-3f42a83d71ac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.322700 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.322698    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7179b107-90f0-47da-991a-3f42a83d71ac-config-out\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.322917 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.322747    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.322917 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.322772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.322917 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.322797    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.407986 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.407939    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:51.424047 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.424013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dldc4\" (UniqueName: \"kubernetes.io/projected/7179b107-90f0-47da-991a-3f42a83d71ac-kube-api-access-dldc4\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.424252 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.424056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-web-config\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.424252 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.424077    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7179b107-90f0-47da-991a-3f42a83d71ac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.424252 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.424121    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7179b107-90f0-47da-991a-3f42a83d71ac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.424252 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.424170    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7179b107-90f0-47da-991a-3f42a83d71ac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.424252 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.424190    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7179b107-90f0-47da-991a-3f42a83d71ac-config-out\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.424252 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.424240    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.424594 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.424264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.424594 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.424287    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.424594 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.424326    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7179b107-90f0-47da-991a-3f42a83d71ac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.424594 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.424378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.424594 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.424413    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-config-volume\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.424594 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.424438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.425593 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.425568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7179b107-90f0-47da-991a-3f42a83d71ac-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.425820 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.425795    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7179b107-90f0-47da-991a-3f42a83d71ac-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.426862 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.426768    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7179b107-90f0-47da-991a-3f42a83d71ac-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.428705 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.428663    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7179b107-90f0-47da-991a-3f42a83d71ac-config-out\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.429419 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.429394    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-web-config\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.430690 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.430634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.431317 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.431293    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7179b107-90f0-47da-991a-3f42a83d71ac-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.431526 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.431444    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-config-volume\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.431526 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.431461    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.431757 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.431733    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.432328 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.432299    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.433806 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.433783    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.435858 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.435833    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dldc4\" (UniqueName: \"kubernetes.io/projected/7179b107-90f0-47da-991a-3f42a83d71ac-kube-api-access-dldc4\") pod \"alertmanager-main-0\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.556966 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.556943    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:16:51.706884 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:51.706854    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:16:51.744949 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:16:51.744921    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7179b107_90f0_47da_991a_3f42a83d71ac.slice/crio-f1cf532dee3ef18484b156d13a9d465dd9156054ee3c34be0d148cbc1462e233 WatchSource:0}: Error finding container f1cf532dee3ef18484b156d13a9d465dd9156054ee3c34be0d148cbc1462e233: Status 404 returned error can't find the container with id f1cf532dee3ef18484b156d13a9d465dd9156054ee3c34be0d148cbc1462e233
Jul 24 21:16:52.272062 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:52.272021    2560 generic.go:356] "Generic (PLEG): container finished" podID="14bf2715-d0e7-48ed-a231-2d6a1737f9bb" containerID="dd12988160a6e4a85b87eed2c4763e1a5f2868d841aa6d181665bfa7d55e8a44" exitCode=0
Jul 24 21:16:52.272534 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:52.272167    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq55j" event={"ID":"14bf2715-d0e7-48ed-a231-2d6a1737f9bb","Type":"ContainerDied","Data":"dd12988160a6e4a85b87eed2c4763e1a5f2868d841aa6d181665bfa7d55e8a44"}
Jul 24 21:16:52.273907 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:52.273880    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7179b107-90f0-47da-991a-3f42a83d71ac","Type":"ContainerStarted","Data":"f1cf532dee3ef18484b156d13a9d465dd9156054ee3c34be0d148cbc1462e233"}
Jul 24 21:16:53.144822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:53.144792    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:53.242341 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:53.242305    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:53.281045 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:53.280860    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw" event={"ID":"dd2f032e-b417-4b44-8208-c885789525df","Type":"ContainerStarted","Data":"70ddffed5fb8f2e488500a777ed04508da19d40a582b982e6850d52f5eded8d7"}
Jul 24 21:16:53.281045 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:53.280978    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw" event={"ID":"dd2f032e-b417-4b44-8208-c885789525df","Type":"ContainerStarted","Data":"99ba8c1551d5476126062e4ba0e0717aac6ff3292558352a4d729254939999cf"}
Jul 24 21:16:53.281045 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:53.281009    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw" event={"ID":"dd2f032e-b417-4b44-8208-c885789525df","Type":"ContainerStarted","Data":"4fe794547b0687f17ad4d772dbbbd5d45dfb9c6040399f261f1d1d18e6ebd7e8"}
Jul 24 21:16:53.285226 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:53.285196    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq55j" event={"ID":"14bf2715-d0e7-48ed-a231-2d6a1737f9bb","Type":"ContainerStarted","Data":"bc15a1a61d4b52174c1090628450e066f15a15d00b89b39b42ac4466e2212781"}
Jul 24 21:16:53.285343 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:53.285245    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qq55j" event={"ID":"14bf2715-d0e7-48ed-a231-2d6a1737f9bb","Type":"ContainerStarted","Data":"5e25aa6e7af4165eaaaa77aff48ddab278716a82233e56e7e49345586562a521"}
Jul 24 21:16:53.286909 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:53.286885    2560 generic.go:356] "Generic (PLEG): container finished" podID="7179b107-90f0-47da-991a-3f42a83d71ac" containerID="5ad0ec3027aeb7d187b83eed44883d17349b35ddf2ecbd2a41787a5e5dbb40c9" exitCode=0
Jul 24 21:16:53.287036 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:53.286953    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7179b107-90f0-47da-991a-3f42a83d71ac","Type":"ContainerDied","Data":"5ad0ec3027aeb7d187b83eed44883d17349b35ddf2ecbd2a41787a5e5dbb40c9"}
Jul 24 21:16:53.301915 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:53.301868    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-mq6zw" podStartSLOduration=1.6848096479999999 podStartE2EDuration="3.301855289s" totalImagesPullingTime="1.617045641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:53.300950268 +0000 UTC m=+92.057296084" watchObservedRunningTime="2026-07-24 21:16:53.301855289 +0000 UTC m=+92.058201103"
Jul 24 21:16:53.319371 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:53.319323    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qq55j" podStartSLOduration=2.179690885 podStartE2EDuration="3.319304524s" totalImagesPullingTime="1.057228563s" totalInitContainerRuntime="82.385076ms" isStatefulPod=true podCreationTimestamp="2026-07-24 21:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:53.31907005 +0000 UTC m=+92.075415874" watchObservedRunningTime="2026-07-24 21:16:53.319304524 +0000 UTC m=+92.075650334"
Jul 24 21:16:53.350026 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:53.349963    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:53.478035 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:53.478003    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:53.685579 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:53.685551    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:54.537243 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.537214    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-54d7c77989-5jbd8"]
Jul 24 21:16:54.540318 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.540299    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.542802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.542782    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 24 21:16:54.543756 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.543738    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 24 21:16:54.543936 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.543919    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 24 21:16:54.543936 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.543929    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7hm10rpspc4q2\""
Jul 24 21:16:54.544064 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.544050    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 24 21:16:54.544156 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.544081    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5fc24\""
Jul 24 21:16:54.556547 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.556523    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54d7c77989-5jbd8"]
Jul 24 21:16:54.653588 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.653563    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jn56\" (UniqueName: \"kubernetes.io/projected/700a1c63-ead1-4c2d-a32f-81070e959ceb-kube-api-access-9jn56\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.653673 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.653606    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/700a1c63-ead1-4c2d-a32f-81070e959ceb-secret-metrics-server-client-certs\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.653719 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.653695    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700a1c63-ead1-4c2d-a32f-81070e959ceb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.653777 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.653760    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/700a1c63-ead1-4c2d-a32f-81070e959ceb-audit-log\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.653815 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.653798    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/700a1c63-ead1-4c2d-a32f-81070e959ceb-client-ca-bundle\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.653910 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.653888    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/700a1c63-ead1-4c2d-a32f-81070e959ceb-metrics-server-audit-profiles\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.654018 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.653929    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/700a1c63-ead1-4c2d-a32f-81070e959ceb-secret-metrics-server-tls\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.754427 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.754372    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/700a1c63-ead1-4c2d-a32f-81070e959ceb-audit-log\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.754427 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.754404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/700a1c63-ead1-4c2d-a32f-81070e959ceb-client-ca-bundle\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.754576 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.754457    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/700a1c63-ead1-4c2d-a32f-81070e959ceb-metrics-server-audit-profiles\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.754576 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.754480    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/700a1c63-ead1-4c2d-a32f-81070e959ceb-secret-metrics-server-tls\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.754576 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.754505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jn56\" (UniqueName: \"kubernetes.io/projected/700a1c63-ead1-4c2d-a32f-81070e959ceb-kube-api-access-9jn56\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.754576 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.754532    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/700a1c63-ead1-4c2d-a32f-81070e959ceb-secret-metrics-server-client-certs\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.754805 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.754579    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700a1c63-ead1-4c2d-a32f-81070e959ceb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.754805 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.754698    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/700a1c63-ead1-4c2d-a32f-81070e959ceb-audit-log\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.755329 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.755308    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/700a1c63-ead1-4c2d-a32f-81070e959ceb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.755447 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.755412    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/700a1c63-ead1-4c2d-a32f-81070e959ceb-metrics-server-audit-profiles\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.757615 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.757589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/700a1c63-ead1-4c2d-a32f-81070e959ceb-client-ca-bundle\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.757702 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.757596    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/700a1c63-ead1-4c2d-a32f-81070e959ceb-secret-metrics-server-client-certs\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.757702 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.757634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/700a1c63-ead1-4c2d-a32f-81070e959ceb-secret-metrics-server-tls\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.763818 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.763796    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jn56\" (UniqueName: \"kubernetes.io/projected/700a1c63-ead1-4c2d-a32f-81070e959ceb-kube-api-access-9jn56\") pod \"metrics-server-54d7c77989-5jbd8\" (UID: \"700a1c63-ead1-4c2d-a32f-81070e959ceb\") " pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.849366 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.849342    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:16:54.916174 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.915757    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:54.991236 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:54.991165    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-54d7c77989-5jbd8"]
Jul 24 21:16:54.993377 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:16:54.993353    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700a1c63_ead1_4c2d_a32f_81070e959ceb.slice/crio-30d6f9f91f5b035649f2a16b0821d73f7af11cd3c643f50386fa50e2c7549755 WatchSource:0}: Error finding container 30d6f9f91f5b035649f2a16b0821d73f7af11cd3c643f50386fa50e2c7549755: Status 404 returned error can't find the container with id 30d6f9f91f5b035649f2a16b0821d73f7af11cd3c643f50386fa50e2c7549755
Jul 24 21:16:55.121151 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:55.121099    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:55.297201 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:55.297163    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7179b107-90f0-47da-991a-3f42a83d71ac","Type":"ContainerStarted","Data":"0d6b2040329759d2e5427955ba7820e5c95a08e2b13ce9b427fcb26ec9fc939b"}
Jul 24 21:16:55.297358 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:55.297235    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7179b107-90f0-47da-991a-3f42a83d71ac","Type":"ContainerStarted","Data":"795b407fe4e8fbf5563651a593b7324f96f3aee99b64c3be899bc3393e227fb4"}
Jul 24 21:16:55.297358 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:55.297279    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7179b107-90f0-47da-991a-3f42a83d71ac","Type":"ContainerStarted","Data":"7051695aff8986cda45eccddc776781fbc0deac6adcf2e1a2903270c61b20ce9"}
Jul 24 21:16:55.297358 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:55.297313    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7179b107-90f0-47da-991a-3f42a83d71ac","Type":"ContainerStarted","Data":"bb1c21085ebc916090df6ec93ecd41fb2332a78bf4e6e2a7c397192fc7be4f01"}
Jul 24 21:16:55.297358 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:55.297341    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7179b107-90f0-47da-991a-3f42a83d71ac","Type":"ContainerStarted","Data":"a587676bc8791ef1afb400a4e2a62e876334d3a1de612f963779190bea591a84"}
Jul 24 21:16:55.298263 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:55.298244    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8" event={"ID":"700a1c63-ead1-4c2d-a32f-81070e959ceb","Type":"ContainerStarted","Data":"30d6f9f91f5b035649f2a16b0821d73f7af11cd3c643f50386fa50e2c7549755"}
Jul 24 21:16:55.759096 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:55.759046    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:55.856532 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:55.856503    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:55.957849 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:55.957534    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:56.058192 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.058163    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:56.220295 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.220208    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:56.306650 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.306621    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7179b107-90f0-47da-991a-3f42a83d71ac","Type":"ContainerStarted","Data":"e4cddf39a2b21dc8c97c81364039186869cdecda87d967b608d9dd66877034ef"}
Jul 24 21:16:56.325818 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.325789    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:56.340222 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.340171    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.190717867 podStartE2EDuration="5.340152639s" totalImagesPullingTime="3.121579916s" totalInitContainerRuntime="27.854856ms" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:56.339696503 +0000 UTC m=+95.096042342" watchObservedRunningTime="2026-07-24 21:16:56.340152639 +0000 UTC m=+95.096498443"
Jul 24 21:16:56.450348 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.450306    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:56.453325 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.453298    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:16:56.457936 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.457909    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.460828 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.460749    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 24 21:16:56.460940 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.460846    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5h8ihnuhrvh18\""
Jul 24 21:16:56.461168 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.461116    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 24 21:16:56.461256 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.461216    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 24 21:16:56.462068 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.462045    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 24 21:16:56.462179 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.462160    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 24 21:16:56.462307 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.462286    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 24 21:16:56.462441 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.462421    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 24 21:16:56.462734 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.462717    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 24 21:16:56.463027 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.463008    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 24 21:16:56.463115 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.463094    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5sv6q\""
Jul 24 21:16:56.463719 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.463226    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 24 21:16:56.463719 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.463426    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 24 21:16:56.463719 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.463555    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 24 21:16:56.463719 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.463643    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 24 21:16:56.474976 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.473648    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 24 21:16:56.479513 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.479491    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:16:56.520443 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.520400    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:56.568951 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.568918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569118 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.568960    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569118 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.568993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569118 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569041    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-config\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569118 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569065    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569118 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569118 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569105    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569195    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569256    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569274    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569290    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569395    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569434    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569467    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569493    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569516    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.569720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.569613    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2744\" (UniqueName: \"kubernetes.io/projected/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-kube-api-access-f2744\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670227 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670399 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670248    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670399 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670291    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670512 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670430    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-config\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670512 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670512 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670507    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670662 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670536    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670662 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670563    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670662 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670596    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670662 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670618    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670662 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670641    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670894 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670894 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670894 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670894 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670894 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670813    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.670894 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.671183 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670947    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.671183 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.670973    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.671183 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.671027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2744\" (UniqueName: \"kubernetes.io/projected/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-kube-api-access-f2744\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.674561 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.673460    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.674561 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.674332    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.675441 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.675327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.676524 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.675966    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.676524 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.676035    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.676524 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.676189    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.676524 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.676430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.676524 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.676473    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-config\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.677880 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.677300    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.677880 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.677368    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.677880 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.677433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.677880 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.677825    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.678200 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.677905    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.678302 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.678282    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.680216 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.680175    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.680710 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.680693    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.681256 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.681170    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.681459 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.681439    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.682711 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.682687    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.682802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.682785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2744\" (UniqueName: \"kubernetes.io/projected/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-kube-api-access-f2744\") pod \"prometheus-k8s-0\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.732368 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.732297    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:56.775697 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.775662    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:16:56.831956 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.831526    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:56.891152 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.889962    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:56.943828 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:56.943795    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:16:56.947293 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:16:56.947261    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c888068_d60f_42eb_9fe1_5325a9f1f9a7.slice/crio-62b8bc0d5575d78a6a31729c6a40686fc8a7ecb6e160d9dd1b18d79a6e73b5aa WatchSource:0}: Error finding container 62b8bc0d5575d78a6a31729c6a40686fc8a7ecb6e160d9dd1b18d79a6e73b5aa: Status 404 returned error can't find the container with id 62b8bc0d5575d78a6a31729c6a40686fc8a7ecb6e160d9dd1b18d79a6e73b5aa
Jul 24 21:16:57.094118 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:57.094085    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:57.312823 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:57.312723    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8" event={"ID":"700a1c63-ead1-4c2d-a32f-81070e959ceb","Type":"ContainerStarted","Data":"4243d4654f67d059f8800b6f5da31549a1a5cf291bfd25653d622ef1bfa46e98"}
Jul 24 21:16:57.314288 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:57.314263    2560 generic.go:356] "Generic (PLEG): container finished" podID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerID="8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed" exitCode=0
Jul 24 21:16:57.314385 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:57.314341    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c888068-d60f-42eb-9fe1-5325a9f1f9a7","Type":"ContainerDied","Data":"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed"}
Jul 24 21:16:57.314432 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:57.314392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c888068-d60f-42eb-9fe1-5325a9f1f9a7","Type":"ContainerStarted","Data":"62b8bc0d5575d78a6a31729c6a40686fc8a7ecb6e160d9dd1b18d79a6e73b5aa"}
Jul 24 21:16:57.332097 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:57.332054    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8" podStartSLOduration=1.975950118 podStartE2EDuration="3.332038253s" totalImagesPullingTime="1.356088135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:16:57.330614147 +0000 UTC m=+96.086959959" watchObservedRunningTime="2026-07-24 21:16:57.332038253 +0000 UTC m=+96.088384065"
Jul 24 21:16:57.433204 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:57.433174    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:57.562546 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:57.562520    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:57.712141 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:57.712101    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:58.497201 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:58.497171    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:58.557138 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:58.557089    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:58.641163 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:58.641115    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:58.744337 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:58.744297    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:59.136170 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:59.136117    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:16:59.200856 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:16:59.200826    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:01.334247 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:01.334211    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c888068-d60f-42eb-9fe1-5325a9f1f9a7","Type":"ContainerStarted","Data":"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"}
Jul 24 21:17:01.334647 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:01.334295    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c888068-d60f-42eb-9fe1-5325a9f1f9a7","Type":"ContainerStarted","Data":"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"}
Jul 24 21:17:03.348919 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:03.348876    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c888068-d60f-42eb-9fe1-5325a9f1f9a7","Type":"ContainerStarted","Data":"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"}
Jul 24 21:17:03.349279 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:03.348979    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c888068-d60f-42eb-9fe1-5325a9f1f9a7","Type":"ContainerStarted","Data":"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"}
Jul 24 21:17:03.349279 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:03.349007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c888068-d60f-42eb-9fe1-5325a9f1f9a7","Type":"ContainerStarted","Data":"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"}
Jul 24 21:17:03.349279 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:03.349050    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c888068-d60f-42eb-9fe1-5325a9f1f9a7","Type":"ContainerStarted","Data":"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"}
Jul 24 21:17:03.386092 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:03.386043    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.519800446 podStartE2EDuration="7.3860039s" totalImagesPullingTime="4.845201831s" totalInitContainerRuntime="21.001623ms" isStatefulPod=false podCreationTimestamp="2026-07-24 21:16:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:17:03.384985755 +0000 UTC m=+102.141331572" watchObservedRunningTime="2026-07-24 21:17:03.3860039 +0000 UTC m=+102.142349712"
Jul 24 21:17:04.475790 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:04.475760    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:04.592425 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:04.592393    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:04.693231 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:04.693198    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:05.187175 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:05.187146    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cpmnw"
Jul 24 21:17:05.827966 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:05.827935    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:05.929181 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:05.929151    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:06.136720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:06.136665    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:06.203671 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:06.203637    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:06.776295 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:06.776269    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:17:07.246286 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:07.246257    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:07.347115 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:07.347088    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:09.823318 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:09.823285    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:09.888642 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:09.888608    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:10.061979 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:10.061945    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:10.118755 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:10.118723    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:11.364678 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:11.364642    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:11.466467 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:11.466434    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:13.044974 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:13.044931    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:13.145021 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:13.144990    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:13.614092 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:13.614065    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:13.676878 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:13.676847    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:14.839879 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:14.839847    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:14.849666 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:14.849646    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:17:14.849733 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:14.849697    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:17:14.904352 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:14.904315    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:15.159609 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:15.159578    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:15.229259 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:15.229227    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:20.652511 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:20.652479    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:20.751612 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:20.751583    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:20.852326 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:20.852293    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:20.922973 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:20.922886    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:21.450957 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:21.450898    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:24.826112 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:24.826077    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:24.901168 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:24.901117    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:31.575485 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:31.575451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs\") pod \"network-metrics-daemon-ldqjs\" (UID: \"de44a92d-8440-4b1f-a0ba-479919425485\") " pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:17:31.577792 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:31.577774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de44a92d-8440-4b1f-a0ba-479919425485-metrics-certs\") pod \"network-metrics-daemon-ldqjs\" (UID: \"de44a92d-8440-4b1f-a0ba-479919425485\") " pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:17:31.679312 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:31.679282    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdtkv\""
Jul 24 21:17:31.686598 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:31.686577    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldqjs"
Jul 24 21:17:32.171447 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:32.171419    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldqjs"]
Jul 24 21:17:32.175429 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:17:32.175404    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde44a92d_8440_4b1f_a0ba_479919425485.slice/crio-f23a2a83ec80094a966a4846066c477ca549e3fe2a4cfc8019d8bbbb1618e947 WatchSource:0}: Error finding container f23a2a83ec80094a966a4846066c477ca549e3fe2a4cfc8019d8bbbb1618e947: Status 404 returned error can't find the container with id f23a2a83ec80094a966a4846066c477ca549e3fe2a4cfc8019d8bbbb1618e947
Jul 24 21:17:32.460350 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:32.460316    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldqjs" event={"ID":"de44a92d-8440-4b1f-a0ba-479919425485","Type":"ContainerStarted","Data":"f23a2a83ec80094a966a4846066c477ca549e3fe2a4cfc8019d8bbbb1618e947"}
Jul 24 21:17:34.474681 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:34.474637    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldqjs" event={"ID":"de44a92d-8440-4b1f-a0ba-479919425485","Type":"ContainerStarted","Data":"84cc279dc8f030fb9c6c4ecb277362756fe20bca87e777784fdbc3d8ba5efc73"}
Jul 24 21:17:34.475184 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:34.474725    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldqjs" event={"ID":"de44a92d-8440-4b1f-a0ba-479919425485","Type":"ContainerStarted","Data":"801018e81cc1ad2a03d9af67848d6ace6a2d9f06d846a93b576b15224a4b2db9"}
Jul 24 21:17:34.491759 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:34.491716    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ldqjs" podStartSLOduration=131.140555708 podStartE2EDuration="2m12.491699825s" totalImagesPullingTime="1.351144117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:17:34.489825139 +0000 UTC m=+133.246170951" watchObservedRunningTime="2026-07-24 21:17:34.491699825 +0000 UTC m=+133.248045638"
Jul 24 21:17:34.593629 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:34.593597    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:34.649951 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:34.649923    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:34.855817 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:34.855749    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:17:34.859571 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:34.859551    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-54d7c77989-5jbd8"
Jul 24 21:17:39.818604 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:39.818574    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:39.933258 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:39.933228    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:17:40.501863 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:40.501826    2560 generic.go:356] "Generic (PLEG): container finished" podID="df093810-8a45-4cc0-b0e2-a1839e46b254" containerID="b3acea82b8154310c86cddfc3b2460b7cfd5646adf40900974427731b6109ff3" exitCode=0
Jul 24 21:17:40.501965 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:40.501870    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj" event={"ID":"df093810-8a45-4cc0-b0e2-a1839e46b254","Type":"ContainerDied","Data":"b3acea82b8154310c86cddfc3b2460b7cfd5646adf40900974427731b6109ff3"}
Jul 24 21:17:40.502182 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:40.502169    2560 scope.go:117] "RemoveContainer" containerID="b3acea82b8154310c86cddfc3b2460b7cfd5646adf40900974427731b6109ff3"
Jul 24 21:17:41.507190 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:41.507153    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-2ffrj" event={"ID":"df093810-8a45-4cc0-b0e2-a1839e46b254","Type":"ContainerStarted","Data":"0856bac4b87c1728fe0e7e2e28dcfd7aa6f87818395570f619559d97674e399d"}
Jul 24 21:17:46.529358 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:46.529326    2560 generic.go:356] "Generic (PLEG): container finished" podID="38bf3087-9101-4b71-be34-dd0726961e2a" containerID="290fc800c282348951d3a292bd70ecc464f610eb79adf5bd8e1e3ee4fb21680a" exitCode=0
Jul 24 21:17:46.529754 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:46.529396    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6" event={"ID":"38bf3087-9101-4b71-be34-dd0726961e2a","Type":"ContainerDied","Data":"290fc800c282348951d3a292bd70ecc464f610eb79adf5bd8e1e3ee4fb21680a"}
Jul 24 21:17:46.529754 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:46.529734    2560 scope.go:117] "RemoveContainer" containerID="290fc800c282348951d3a292bd70ecc464f610eb79adf5bd8e1e3ee4fb21680a"
Jul 24 21:17:47.536031 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:47.536001    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6" event={"ID":"38bf3087-9101-4b71-be34-dd0726961e2a","Type":"ContainerStarted","Data":"333e29cc5d18a14c00fa9e38db25ce2f9b9b1803c9772664e2fa8e2f7bcf8002"}
Jul 24 21:17:56.776344 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:56.776313    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:17:56.796332 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:56.796309    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:17:57.591451 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:17:57.591425    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:09.431556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:09.431520    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:09.634779 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:09.634749    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:10.488515 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:10.488478    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:18:10.488913 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:10.488845    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="alertmanager" containerID="cri-o://a587676bc8791ef1afb400a4e2a62e876334d3a1de612f963779190bea591a84" gracePeriod=120
Jul 24 21:18:10.488995 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:10.488905    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="kube-rbac-proxy-metric" containerID="cri-o://0d6b2040329759d2e5427955ba7820e5c95a08e2b13ce9b427fcb26ec9fc939b" gracePeriod=120
Jul 24 21:18:10.488995 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:10.488921    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="kube-rbac-proxy-web" containerID="cri-o://7051695aff8986cda45eccddc776781fbc0deac6adcf2e1a2903270c61b20ce9" gracePeriod=120
Jul 24 21:18:10.488995 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:10.488953    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="kube-rbac-proxy" containerID="cri-o://795b407fe4e8fbf5563651a593b7324f96f3aee99b64c3be899bc3393e227fb4" gracePeriod=120
Jul 24 21:18:10.489173 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:10.488978    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="prom-label-proxy" containerID="cri-o://e4cddf39a2b21dc8c97c81364039186869cdecda87d967b608d9dd66877034ef" gracePeriod=120
Jul 24 21:18:10.489173 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:10.489056    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="config-reloader" containerID="cri-o://bb1c21085ebc916090df6ec93ecd41fb2332a78bf4e6e2a7c397192fc7be4f01" gracePeriod=120
Jul 24 21:18:10.628512 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:10.628486    2560 generic.go:356] "Generic (PLEG): container finished" podID="7179b107-90f0-47da-991a-3f42a83d71ac" containerID="e4cddf39a2b21dc8c97c81364039186869cdecda87d967b608d9dd66877034ef" exitCode=0
Jul 24 21:18:10.628512 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:10.628507    2560 generic.go:356] "Generic (PLEG): container finished" podID="7179b107-90f0-47da-991a-3f42a83d71ac" containerID="795b407fe4e8fbf5563651a593b7324f96f3aee99b64c3be899bc3393e227fb4" exitCode=0
Jul 24 21:18:10.628512 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:10.628514    2560 generic.go:356] "Generic (PLEG): container finished" podID="7179b107-90f0-47da-991a-3f42a83d71ac" containerID="bb1c21085ebc916090df6ec93ecd41fb2332a78bf4e6e2a7c397192fc7be4f01" exitCode=0
Jul 24 21:18:10.628669 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:10.628520    2560 generic.go:356] "Generic (PLEG): container finished" podID="7179b107-90f0-47da-991a-3f42a83d71ac" containerID="a587676bc8791ef1afb400a4e2a62e876334d3a1de612f963779190bea591a84" exitCode=0
Jul 24 21:18:10.628669 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:10.628555    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7179b107-90f0-47da-991a-3f42a83d71ac","Type":"ContainerDied","Data":"e4cddf39a2b21dc8c97c81364039186869cdecda87d967b608d9dd66877034ef"}
Jul 24 21:18:10.628669 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:10.628623    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7179b107-90f0-47da-991a-3f42a83d71ac","Type":"ContainerDied","Data":"795b407fe4e8fbf5563651a593b7324f96f3aee99b64c3be899bc3393e227fb4"}
Jul 24 21:18:10.628669 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:10.628651    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7179b107-90f0-47da-991a-3f42a83d71ac","Type":"ContainerDied","Data":"bb1c21085ebc916090df6ec93ecd41fb2332a78bf4e6e2a7c397192fc7be4f01"}
Jul 24 21:18:10.628790 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:10.628677    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7179b107-90f0-47da-991a-3f42a83d71ac","Type":"ContainerDied","Data":"a587676bc8791ef1afb400a4e2a62e876334d3a1de612f963779190bea591a84"}
Jul 24 21:18:11.636744 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.636708    2560 generic.go:356] "Generic (PLEG): container finished" podID="7179b107-90f0-47da-991a-3f42a83d71ac" containerID="0d6b2040329759d2e5427955ba7820e5c95a08e2b13ce9b427fcb26ec9fc939b" exitCode=0
Jul 24 21:18:11.636744 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.636740    2560 generic.go:356] "Generic (PLEG): container finished" podID="7179b107-90f0-47da-991a-3f42a83d71ac" containerID="7051695aff8986cda45eccddc776781fbc0deac6adcf2e1a2903270c61b20ce9" exitCode=0
Jul 24 21:18:11.637118 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.636738    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7179b107-90f0-47da-991a-3f42a83d71ac","Type":"ContainerDied","Data":"0d6b2040329759d2e5427955ba7820e5c95a08e2b13ce9b427fcb26ec9fc939b"}
Jul 24 21:18:11.637118 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.636831    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7179b107-90f0-47da-991a-3f42a83d71ac","Type":"ContainerDied","Data":"7051695aff8986cda45eccddc776781fbc0deac6adcf2e1a2903270c61b20ce9"}
Jul 24 21:18:11.735029 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.735008    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:11.812859 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.812748    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7179b107-90f0-47da-991a-3f42a83d71ac\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") "
Jul 24 21:18:11.812859 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.812807    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-web-config\") pod \"7179b107-90f0-47da-991a-3f42a83d71ac\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") "
Jul 24 21:18:11.813079 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.812903    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dldc4\" (UniqueName: \"kubernetes.io/projected/7179b107-90f0-47da-991a-3f42a83d71ac-kube-api-access-dldc4\") pod \"7179b107-90f0-47da-991a-3f42a83d71ac\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") "
Jul 24 21:18:11.813079 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.812943    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy-web\") pod \"7179b107-90f0-47da-991a-3f42a83d71ac\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") "
Jul 24 21:18:11.813079 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.812973    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-main-tls\") pod \"7179b107-90f0-47da-991a-3f42a83d71ac\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") "
Jul 24 21:18:11.813079 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.813013    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-config-volume\") pod \"7179b107-90f0-47da-991a-3f42a83d71ac\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") "
Jul 24 21:18:11.813079 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.813047    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7179b107-90f0-47da-991a-3f42a83d71ac-tls-assets\") pod \"7179b107-90f0-47da-991a-3f42a83d71ac\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") "
Jul 24 21:18:11.813079 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.813070    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-cluster-tls-config\") pod \"7179b107-90f0-47da-991a-3f42a83d71ac\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") "
Jul 24 21:18:11.813472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.813102    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7179b107-90f0-47da-991a-3f42a83d71ac-alertmanager-main-db\") pod \"7179b107-90f0-47da-991a-3f42a83d71ac\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") "
Jul 24 21:18:11.813472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.813169    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7179b107-90f0-47da-991a-3f42a83d71ac-alertmanager-trusted-ca-bundle\") pod \"7179b107-90f0-47da-991a-3f42a83d71ac\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") "
Jul 24 21:18:11.813472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.813244    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7179b107-90f0-47da-991a-3f42a83d71ac-config-out\") pod \"7179b107-90f0-47da-991a-3f42a83d71ac\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") "
Jul 24 21:18:11.813472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.813275    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7179b107-90f0-47da-991a-3f42a83d71ac-metrics-client-ca\") pod \"7179b107-90f0-47da-991a-3f42a83d71ac\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") "
Jul 24 21:18:11.813472 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.813300    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy\") pod \"7179b107-90f0-47da-991a-3f42a83d71ac\" (UID: \"7179b107-90f0-47da-991a-3f42a83d71ac\") "
Jul 24 21:18:11.813723 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.813692    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7179b107-90f0-47da-991a-3f42a83d71ac-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "7179b107-90f0-47da-991a-3f42a83d71ac" (UID: "7179b107-90f0-47da-991a-3f42a83d71ac"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:18:11.814474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.814048    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7179b107-90f0-47da-991a-3f42a83d71ac-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "7179b107-90f0-47da-991a-3f42a83d71ac" (UID: "7179b107-90f0-47da-991a-3f42a83d71ac"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:11.814474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.814450    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7179b107-90f0-47da-991a-3f42a83d71ac-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "7179b107-90f0-47da-991a-3f42a83d71ac" (UID: "7179b107-90f0-47da-991a-3f42a83d71ac"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:11.816403 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.816345    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7179b107-90f0-47da-991a-3f42a83d71ac-kube-api-access-dldc4" (OuterVolumeSpecName: "kube-api-access-dldc4") pod "7179b107-90f0-47da-991a-3f42a83d71ac" (UID: "7179b107-90f0-47da-991a-3f42a83d71ac"). InnerVolumeSpecName "kube-api-access-dldc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:18:11.817420 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.817388    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "7179b107-90f0-47da-991a-3f42a83d71ac" (UID: "7179b107-90f0-47da-991a-3f42a83d71ac"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:11.817685 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.817634    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "7179b107-90f0-47da-991a-3f42a83d71ac" (UID: "7179b107-90f0-47da-991a-3f42a83d71ac"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:11.817685 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.817650    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "7179b107-90f0-47da-991a-3f42a83d71ac" (UID: "7179b107-90f0-47da-991a-3f42a83d71ac"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:11.817837 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.817746    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7179b107-90f0-47da-991a-3f42a83d71ac-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7179b107-90f0-47da-991a-3f42a83d71ac" (UID: "7179b107-90f0-47da-991a-3f42a83d71ac"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:18:11.817837 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.817790    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7179b107-90f0-47da-991a-3f42a83d71ac-config-out" (OuterVolumeSpecName: "config-out") pod "7179b107-90f0-47da-991a-3f42a83d71ac" (UID: "7179b107-90f0-47da-991a-3f42a83d71ac"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:18:11.818091 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.818071    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-config-volume" (OuterVolumeSpecName: "config-volume") pod "7179b107-90f0-47da-991a-3f42a83d71ac" (UID: "7179b107-90f0-47da-991a-3f42a83d71ac"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:11.818350 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.818334    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "7179b107-90f0-47da-991a-3f42a83d71ac" (UID: "7179b107-90f0-47da-991a-3f42a83d71ac"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:11.822334 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.822310    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "7179b107-90f0-47da-991a-3f42a83d71ac" (UID: "7179b107-90f0-47da-991a-3f42a83d71ac"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:11.828265 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.828245    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-web-config" (OuterVolumeSpecName: "web-config") pod "7179b107-90f0-47da-991a-3f42a83d71ac" (UID: "7179b107-90f0-47da-991a-3f42a83d71ac"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:11.914788 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.914758    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7179b107-90f0-47da-991a-3f42a83d71ac-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:11.914788 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.914781    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7179b107-90f0-47da-991a-3f42a83d71ac-config-out\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:11.914788 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.914791    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7179b107-90f0-47da-991a-3f42a83d71ac-metrics-client-ca\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:11.915010 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.914801    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:11.915010 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.914810    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:11.915010 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.914819    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-web-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:11.915010 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.914828    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dldc4\" (UniqueName: \"kubernetes.io/projected/7179b107-90f0-47da-991a-3f42a83d71ac-kube-api-access-dldc4\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:11.915010 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.914836    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:11.915010 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.914844    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-secret-alertmanager-main-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:11.915010 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.914853    2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-config-volume\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:11.915010 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.914861    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7179b107-90f0-47da-991a-3f42a83d71ac-tls-assets\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:11.915010 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.914868    2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7179b107-90f0-47da-991a-3f42a83d71ac-cluster-tls-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:11.915010 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:11.914876    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7179b107-90f0-47da-991a-3f42a83d71ac-alertmanager-main-db\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:12.644076 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:12.644042    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7179b107-90f0-47da-991a-3f42a83d71ac","Type":"ContainerDied","Data":"f1cf532dee3ef18484b156d13a9d465dd9156054ee3c34be0d148cbc1462e233"}
Jul 24 21:18:12.644076 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:12.644062    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 24 21:18:12.644513 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:12.644169    2560 scope.go:117] "RemoveContainer" containerID="e4cddf39a2b21dc8c97c81364039186869cdecda87d967b608d9dd66877034ef"
Jul 24 21:18:12.652962 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:12.652936    2560 scope.go:117] "RemoveContainer" containerID="0d6b2040329759d2e5427955ba7820e5c95a08e2b13ce9b427fcb26ec9fc939b"
Jul 24 21:18:12.659947 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:12.659931    2560 scope.go:117] "RemoveContainer" containerID="795b407fe4e8fbf5563651a593b7324f96f3aee99b64c3be899bc3393e227fb4"
Jul 24 21:18:12.666685 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:12.666664    2560 scope.go:117] "RemoveContainer" containerID="7051695aff8986cda45eccddc776781fbc0deac6adcf2e1a2903270c61b20ce9"
Jul 24 21:18:12.669643 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:12.669621    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:18:12.673732 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:12.673708    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 24 21:18:12.675300 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:12.675287    2560 scope.go:117] "RemoveContainer" containerID="bb1c21085ebc916090df6ec93ecd41fb2332a78bf4e6e2a7c397192fc7be4f01"
Jul 24 21:18:12.682350 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:12.682330    2560 scope.go:117] "RemoveContainer" containerID="a587676bc8791ef1afb400a4e2a62e876334d3a1de612f963779190bea591a84"
Jul 24 21:18:12.689375 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:12.689357    2560 scope.go:117] "RemoveContainer" containerID="5ad0ec3027aeb7d187b83eed44883d17349b35ddf2ecbd2a41787a5e5dbb40c9"
Jul 24 21:18:13.753682 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:13.753653    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" path="/var/lib/kubelet/pods/7179b107-90f0-47da-991a-3f42a83d71ac/volumes"
Jul 24 21:18:14.795080 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:14.795046    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:18:14.795488 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:14.795415    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="prometheus" containerID="cri-o://5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9" gracePeriod=600
Jul 24 21:18:14.795548 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:14.795481    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="kube-rbac-proxy-thanos" containerID="cri-o://55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed" gracePeriod=600
Jul 24 21:18:14.795548 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:14.795516    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="thanos-sidecar" containerID="cri-o://1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3" gracePeriod=600
Jul 24 21:18:14.795653 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:14.795566    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="config-reloader" containerID="cri-o://f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07" gracePeriod=600
Jul 24 21:18:14.795653 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:14.795501    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="kube-rbac-proxy" containerID="cri-o://2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20" gracePeriod=600
Jul 24 21:18:14.795746 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:14.795492    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="kube-rbac-proxy-web" containerID="cri-o://3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4" gracePeriod=600
Jul 24 21:18:15.057505 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.057481    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.143516 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.143490    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-trusted-ca-bundle\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.143516 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.143520    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-0\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.143756 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.143540    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2744\" (UniqueName: \"kubernetes.io/projected/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-kube-api-access-f2744\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.143756 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.143560    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-metrics-client-certs\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.143756 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.143585    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.143756 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.143621    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-metrics-client-ca\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.143756 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.143645    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.143756 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.143674    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-1\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.144070 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.143923    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:15.144070 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144030    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-grpc-tls\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.144070 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144043    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:15.144070 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144051    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:15.144070 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144069    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-config\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.144522 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144095    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-thanos-prometheus-http-client-file\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.144522 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144151    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-web-config\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.144522 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144181    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-kube-rbac-proxy\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.144522 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144215    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-config-out\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.144522 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144246    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-kubelet-serving-ca-bundle\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.144522 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144273    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-db\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.144522 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144298    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-tls\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.144522 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144322    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-tls-assets\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.144522 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144354    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-2\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.144522 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144387    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-serving-certs-ca-bundle\") pod \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\" (UID: \"3c888068-d60f-42eb-9fe1-5325a9f1f9a7\") "
Jul 24 21:18:15.145016 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144791    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-metrics-client-ca\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.145016 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144812    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.145016 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144828    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.145016 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.144828    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:15.145801 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.145768    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:15.146904 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.146873    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:15.146904 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.146891    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:18:15.147076 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.147055    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-kube-api-access-f2744" (OuterVolumeSpecName: "kube-api-access-f2744") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "kube-api-access-f2744". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:18:15.147527 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.147197    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:15.147527 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.147333    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:18:15.147837 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.147813    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-config" (OuterVolumeSpecName: "config") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:15.148182 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.148156    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:15.148789 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.148721    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:18:15.148883 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.148829    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:15.148883 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.148856    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:15.149091 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.149069    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:15.149245 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.149219    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:15.149501 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.149480    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-config-out" (OuterVolumeSpecName: "config-out") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:18:15.149647 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.149630    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:15.158709 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.158685    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-web-config" (OuterVolumeSpecName: "web-config") pod "3c888068-d60f-42eb-9fe1-5325a9f1f9a7" (UID: "3c888068-d60f-42eb-9fe1-5325a9f1f9a7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:18:15.245990 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.245958    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.245990 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.245986    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-db\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.245990 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.245998    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.246249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.246008    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-tls-assets\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.246249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.246018    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.246249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.246026    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.246249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.246035    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.246249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.246045    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2744\" (UniqueName: \"kubernetes.io/projected/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-kube-api-access-f2744\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.246249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.246053    2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-metrics-client-certs\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.246249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.246061    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.246249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.246070    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.246249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.246079    2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-grpc-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.246249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.246086    2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.246249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.246094    2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.246249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.246102    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-web-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.246249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.246109    2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-secret-kube-rbac-proxy\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.246249 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.246116    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3c888068-d60f-42eb-9fe1-5325a9f1f9a7-config-out\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:18:15.658985 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.658949    2560 generic.go:356] "Generic (PLEG): container finished" podID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerID="55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed" exitCode=0
Jul 24 21:18:15.658985 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.658975    2560 generic.go:356] "Generic (PLEG): container finished" podID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerID="2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20" exitCode=0
Jul 24 21:18:15.658985 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.658983    2560 generic.go:356] "Generic (PLEG): container finished" podID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerID="3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4" exitCode=0
Jul 24 21:18:15.658985 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.658990    2560 generic.go:356] "Generic (PLEG): container finished" podID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerID="1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3" exitCode=0
Jul 24 21:18:15.659381 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.658998    2560 generic.go:356] "Generic (PLEG): container finished" podID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerID="f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07" exitCode=0
Jul 24 21:18:15.659381 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.659007    2560 generic.go:356] "Generic (PLEG): container finished" podID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerID="5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9" exitCode=0
Jul 24 21:18:15.659381 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.659036    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.659381 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.659034    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c888068-d60f-42eb-9fe1-5325a9f1f9a7","Type":"ContainerDied","Data":"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"}
Jul 24 21:18:15.659381 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.659190    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c888068-d60f-42eb-9fe1-5325a9f1f9a7","Type":"ContainerDied","Data":"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"}
Jul 24 21:18:15.659381 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.659221    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c888068-d60f-42eb-9fe1-5325a9f1f9a7","Type":"ContainerDied","Data":"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"}
Jul 24 21:18:15.659381 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.659247    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c888068-d60f-42eb-9fe1-5325a9f1f9a7","Type":"ContainerDied","Data":"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"}
Jul 24 21:18:15.659381 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.659258    2560 scope.go:117] "RemoveContainer" containerID="55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"
Jul 24 21:18:15.659381 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.659270    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c888068-d60f-42eb-9fe1-5325a9f1f9a7","Type":"ContainerDied","Data":"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"}
Jul 24 21:18:15.659381 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.659293    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c888068-d60f-42eb-9fe1-5325a9f1f9a7","Type":"ContainerDied","Data":"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"}
Jul 24 21:18:15.659381 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.659317    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3c888068-d60f-42eb-9fe1-5325a9f1f9a7","Type":"ContainerDied","Data":"62b8bc0d5575d78a6a31729c6a40686fc8a7ecb6e160d9dd1b18d79a6e73b5aa"}
Jul 24 21:18:15.670007 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.669987    2560 scope.go:117] "RemoveContainer" containerID="2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"
Jul 24 21:18:15.677284 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.677268    2560 scope.go:117] "RemoveContainer" containerID="3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"
Jul 24 21:18:15.686097 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.686082    2560 scope.go:117] "RemoveContainer" containerID="1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"
Jul 24 21:18:15.689530 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.689507    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:18:15.693634 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.693612    2560 scope.go:117] "RemoveContainer" containerID="f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"
Jul 24 21:18:15.694791 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.694775    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:18:15.700220 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.700204    2560 scope.go:117] "RemoveContainer" containerID="5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"
Jul 24 21:18:15.707314 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.707299    2560 scope.go:117] "RemoveContainer" containerID="8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed"
Jul 24 21:18:15.713850 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.713829    2560 scope.go:117] "RemoveContainer" containerID="55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"
Jul 24 21:18:15.714162 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:18:15.714140    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed\": container with ID starting with 55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed not found: ID does not exist" containerID="55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"
Jul 24 21:18:15.714220 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.714170    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"} err="failed to get container status \"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed\": rpc error: code = NotFound desc = could not find container \"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed\": container with ID starting with 55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed not found: ID does not exist"
Jul 24 21:18:15.714220 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.714214    2560 scope.go:117] "RemoveContainer" containerID="2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"
Jul 24 21:18:15.714457 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:18:15.714442    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20\": container with ID starting with 2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20 not found: ID does not exist" containerID="2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"
Jul 24 21:18:15.714519 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.714460    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"} err="failed to get container status \"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20\": rpc error: code = NotFound desc = could not find container \"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20\": container with ID starting with 2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20 not found: ID does not exist"
Jul 24 21:18:15.714519 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.714473    2560 scope.go:117] "RemoveContainer" containerID="3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"
Jul 24 21:18:15.714693 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:18:15.714676    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4\": container with ID starting with 3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4 not found: ID does not exist" containerID="3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"
Jul 24 21:18:15.714731 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.714699    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"} err="failed to get container status \"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4\": rpc error: code = NotFound desc = could not find container \"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4\": container with ID starting with 3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4 not found: ID does not exist"
Jul 24 21:18:15.714731 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.714714    2560 scope.go:117] "RemoveContainer" containerID="1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"
Jul 24 21:18:15.714941 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:18:15.714924    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3\": container with ID starting with 1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3 not found: ID does not exist" containerID="1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"
Jul 24 21:18:15.714986 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.714945    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"} err="failed to get container status \"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3\": rpc error: code = NotFound desc = could not find container \"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3\": container with ID starting with 1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3 not found: ID does not exist"
Jul 24 21:18:15.714986 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.714957    2560 scope.go:117] "RemoveContainer" containerID="f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"
Jul 24 21:18:15.715206 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:18:15.715187    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07\": container with ID starting with f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07 not found: ID does not exist" containerID="f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"
Jul 24 21:18:15.715267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.715213    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"} err="failed to get container status \"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07\": rpc error: code = NotFound desc = could not find container \"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07\": container with ID starting with f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07 not found: ID does not exist"
Jul 24 21:18:15.715267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.715233    2560 scope.go:117] "RemoveContainer" containerID="5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"
Jul 24 21:18:15.715444 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:18:15.715427    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9\": container with ID starting with 5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9 not found: ID does not exist" containerID="5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"
Jul 24 21:18:15.715480 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.715449    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"} err="failed to get container status \"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9\": rpc error: code = NotFound desc = could not find container \"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9\": container with ID starting with 5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9 not found: ID does not exist"
Jul 24 21:18:15.715480 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.715462    2560 scope.go:117] "RemoveContainer" containerID="8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed"
Jul 24 21:18:15.715651 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:18:15.715633    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed\": container with ID starting with 8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed not found: ID does not exist" containerID="8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed"
Jul 24 21:18:15.715712 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.715658    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed"} err="failed to get container status \"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed\": rpc error: code = NotFound desc = could not find container \"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed\": container with ID starting with 8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed not found: ID does not exist"
Jul 24 21:18:15.715712 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.715679    2560 scope.go:117] "RemoveContainer" containerID="55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"
Jul 24 21:18:15.715879 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.715863    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"} err="failed to get container status \"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed\": rpc error: code = NotFound desc = could not find container \"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed\": container with ID starting with 55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed not found: ID does not exist"
Jul 24 21:18:15.715927 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.715880    2560 scope.go:117] "RemoveContainer" containerID="2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"
Jul 24 21:18:15.716050 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.716035    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"} err="failed to get container status \"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20\": rpc error: code = NotFound desc = could not find container \"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20\": container with ID starting with 2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20 not found: ID does not exist"
Jul 24 21:18:15.716097 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.716051    2560 scope.go:117] "RemoveContainer" containerID="3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"
Jul 24 21:18:15.716266 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.716249    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"} err="failed to get container status \"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4\": rpc error: code = NotFound desc = could not find container \"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4\": container with ID starting with 3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4 not found: ID does not exist"
Jul 24 21:18:15.716339 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.716268    2560 scope.go:117] "RemoveContainer" containerID="1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"
Jul 24 21:18:15.716457 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.716441    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"} err="failed to get container status \"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3\": rpc error: code = NotFound desc = could not find container \"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3\": container with ID starting with 1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3 not found: ID does not exist"
Jul 24 21:18:15.716498 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.716458    2560 scope.go:117] "RemoveContainer" containerID="f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"
Jul 24 21:18:15.716679 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.716663    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"} err="failed to get container status \"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07\": rpc error: code = NotFound desc = could not find container \"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07\": container with ID starting with f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07 not found: ID does not exist"
Jul 24 21:18:15.716742 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.716680    2560 scope.go:117] "RemoveContainer" containerID="5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"
Jul 24 21:18:15.716887 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.716872    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"} err="failed to get container status \"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9\": rpc error: code = NotFound desc = could not find container \"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9\": container with ID starting with 5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9 not found: ID does not exist"
Jul 24 21:18:15.716929 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.716888    2560 scope.go:117] "RemoveContainer" containerID="8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed"
Jul 24 21:18:15.717090 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.717070    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed"} err="failed to get container status \"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed\": rpc error: code = NotFound desc = could not find container \"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed\": container with ID starting with 8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed not found: ID does not exist"
Jul 24 21:18:15.717152 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.717091    2560 scope.go:117] "RemoveContainer" containerID="55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"
Jul 24 21:18:15.717307 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.717290    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"} err="failed to get container status \"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed\": rpc error: code = NotFound desc = could not find container \"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed\": container with ID starting with 55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed not found: ID does not exist"
Jul 24 21:18:15.717376 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.717309    2560 scope.go:117] "RemoveContainer" containerID="2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"
Jul 24 21:18:15.717482 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.717466    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"} err="failed to get container status \"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20\": rpc error: code = NotFound desc = could not find container \"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20\": container with ID starting with 2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20 not found: ID does not exist"
Jul 24 21:18:15.717525 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.717485    2560 scope.go:117] "RemoveContainer" containerID="3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"
Jul 24 21:18:15.717707 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.717692    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"} err="failed to get container status \"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4\": rpc error: code = NotFound desc = could not find container \"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4\": container with ID starting with 3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4 not found: ID does not exist"
Jul 24 21:18:15.717750 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.717708    2560 scope.go:117] "RemoveContainer" containerID="1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"
Jul 24 21:18:15.717903 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.717885    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"} err="failed to get container status \"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3\": rpc error: code = NotFound desc = could not find container \"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3\": container with ID starting with 1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3 not found: ID does not exist"
Jul 24 21:18:15.717972 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.717904    2560 scope.go:117] "RemoveContainer" containerID="f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"
Jul 24 21:18:15.718088 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.718073    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"} err="failed to get container status \"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07\": rpc error: code = NotFound desc = could not find container \"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07\": container with ID starting with f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07 not found: ID does not exist"
Jul 24 21:18:15.718165 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.718089    2560 scope.go:117] "RemoveContainer" containerID="5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"
Jul 24 21:18:15.718297 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.718282    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"} err="failed to get container status \"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9\": rpc error: code = NotFound desc = could not find container \"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9\": container with ID starting with 5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9 not found: ID does not exist"
Jul 24 21:18:15.718339 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.718299    2560 scope.go:117] "RemoveContainer" containerID="8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed"
Jul 24 21:18:15.718486 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.718469    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed"} err="failed to get container status \"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed\": rpc error: code = NotFound desc = could not find container \"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed\": container with ID starting with 8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed not found: ID does not exist"
Jul 24 21:18:15.718554 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.718487    2560 scope.go:117] "RemoveContainer" containerID="55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"
Jul 24 21:18:15.718694 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.718664    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"} err="failed to get container status \"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed\": rpc error: code = NotFound desc = could not find container \"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed\": container with ID starting with 55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed not found: ID does not exist"
Jul 24 21:18:15.718694 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.718686    2560 scope.go:117] "RemoveContainer" containerID="2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"
Jul 24 21:18:15.718862 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.718851    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"} err="failed to get container status \"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20\": rpc error: code = NotFound desc = could not find container \"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20\": container with ID starting with 2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20 not found: ID does not exist"
Jul 24 21:18:15.718909 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.718863    2560 scope.go:117] "RemoveContainer" containerID="3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"
Jul 24 21:18:15.719043 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.719027    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"} err="failed to get container status \"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4\": rpc error: code = NotFound desc = could not find container \"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4\": container with ID starting with 3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4 not found: ID does not exist"
Jul 24 21:18:15.719108 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.719045    2560 scope.go:117] "RemoveContainer" containerID="1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"
Jul 24 21:18:15.719299 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.719281    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"} err="failed to get container status \"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3\": rpc error: code = NotFound desc = could not find container \"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3\": container with ID starting with 1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3 not found: ID does not exist"
Jul 24 21:18:15.719348 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.719300    2560 scope.go:117] "RemoveContainer" containerID="f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"
Jul 24 21:18:15.719480 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.719463    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"} err="failed to get container status \"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07\": rpc error: code = NotFound desc = could not find container \"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07\": container with ID starting with f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07 not found: ID does not exist"
Jul 24 21:18:15.719519 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.719481    2560 scope.go:117] "RemoveContainer" containerID="5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"
Jul 24 21:18:15.719667 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.719648    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"} err="failed to get container status \"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9\": rpc error: code = NotFound desc = could not find container \"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9\": container with ID starting with 5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9 not found: ID does not exist"
Jul 24 21:18:15.719735 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.719669    2560 scope.go:117] "RemoveContainer" containerID="8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed"
Jul 24 21:18:15.719855 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.719841    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed"} err="failed to get container status \"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed\": rpc error: code = NotFound desc = could not find container \"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed\": container with ID starting with 8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed not found: ID does not exist"
Jul 24 21:18:15.719898 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.719856    2560 scope.go:117] "RemoveContainer" containerID="55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"
Jul 24 21:18:15.720025 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.720007    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"} err="failed to get container status \"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed\": rpc error: code = NotFound desc = could not find container \"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed\": container with ID starting with 55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed not found: ID does not exist"
Jul 24 21:18:15.720084 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.720027    2560 scope.go:117] "RemoveContainer" containerID="2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"
Jul 24 21:18:15.720235 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.720213    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"} err="failed to get container status \"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20\": rpc error: code = NotFound desc = could not find container \"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20\": container with ID starting with 2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20 not found: ID does not exist"
Jul 24 21:18:15.720290 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.720236    2560 scope.go:117] "RemoveContainer" containerID="3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"
Jul 24 21:18:15.720435 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.720419    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"} err="failed to get container status \"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4\": rpc error: code = NotFound desc = could not find container \"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4\": container with ID starting with 3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4 not found: ID does not exist"
Jul 24 21:18:15.720513 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.720439    2560 scope.go:117] "RemoveContainer" containerID="1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"
Jul 24 21:18:15.720644 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.720627    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"} err="failed to get container status \"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3\": rpc error: code = NotFound desc = could not find container \"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3\": container with ID starting with 1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3 not found: ID does not exist"
Jul 24 21:18:15.720711 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.720645    2560 scope.go:117] "RemoveContainer" containerID="f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"
Jul 24 21:18:15.720847 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.720830    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"} err="failed to get container status \"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07\": rpc error: code = NotFound desc = could not find container \"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07\": container with ID starting with f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07 not found: ID does not exist"
Jul 24 21:18:15.720906 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.720848    2560 scope.go:117] "RemoveContainer" containerID="5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"
Jul 24 21:18:15.721051 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.721033    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"} err="failed to get container status \"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9\": rpc error: code = NotFound desc = could not find container \"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9\": container with ID starting with 5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9 not found: ID does not exist"
Jul 24 21:18:15.721099 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.721053    2560 scope.go:117] "RemoveContainer" containerID="8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed"
Jul 24 21:18:15.721304 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.721288    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed"} err="failed to get container status \"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed\": rpc error: code = NotFound desc = could not find container \"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed\": container with ID starting with 8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed not found: ID does not exist"
Jul 24 21:18:15.721359 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.721305    2560 scope.go:117] "RemoveContainer" containerID="55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"
Jul 24 21:18:15.721476 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.721460    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed"} err="failed to get container status \"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed\": rpc error: code = NotFound desc = could not find container \"55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed\": container with ID starting with 55f1fba0d3344454353a1776daa25b35ec220243355fba2a171cd4068015deed not found: ID does not exist"
Jul 24 21:18:15.721513 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.721477    2560 scope.go:117] "RemoveContainer" containerID="2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"
Jul 24 21:18:15.721637 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.721622    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20"} err="failed to get container status \"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20\": rpc error: code = NotFound desc = could not find container \"2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20\": container with ID starting with 2adc4af931595d4a5d5e3ba63679b65a60358316627d0548142ae540eb5b7b20 not found: ID does not exist"
Jul 24 21:18:15.721678 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.721637    2560 scope.go:117] "RemoveContainer" containerID="3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"
Jul 24 21:18:15.721794 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.721779    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4"} err="failed to get container status \"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4\": rpc error: code = NotFound desc = could not find container \"3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4\": container with ID starting with 3bdfa88e0962c5a0bc31626cd872a3ebd97d8e8834cb4dfb3bdf91101b5a64c4 not found: ID does not exist"
Jul 24 21:18:15.721831 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.721794    2560 scope.go:117] "RemoveContainer" containerID="1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"
Jul 24 21:18:15.721952 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.721937    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3"} err="failed to get container status \"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3\": rpc error: code = NotFound desc = could not find container \"1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3\": container with ID starting with 1da64f101a80f5cc85d2aee9abdf1af948958e529923a270f2e0d456b1e455a3 not found: ID does not exist"
Jul 24 21:18:15.721992 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.721952    2560 scope.go:117] "RemoveContainer" containerID="f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"
Jul 24 21:18:15.722112 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.722098    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07"} err="failed to get container status \"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07\": rpc error: code = NotFound desc = could not find container \"f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07\": container with ID starting with f2061b43de4ba78643576f53c83097cec3c28494f98e7f60d972340c557d1f07 not found: ID does not exist"
Jul 24 21:18:15.722171 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.722113    2560 scope.go:117] "RemoveContainer" containerID="5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"
Jul 24 21:18:15.722295 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.722279    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9"} err="failed to get container status \"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9\": rpc error: code = NotFound desc = could not find container \"5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9\": container with ID starting with 5d9982110d5453d51fdd78c357f68c5771e9426c21f9282c1408bc0dcb6654d9 not found: ID does not exist"
Jul 24 21:18:15.722336 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.722295    2560 scope.go:117] "RemoveContainer" containerID="8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed"
Jul 24 21:18:15.722466 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.722449    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed"} err="failed to get container status \"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed\": rpc error: code = NotFound desc = could not find container \"8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed\": container with ID starting with 8c8164d5be97d4f92cfbfc9f02b64e668a002224abcaf333227930b0179bd0ed not found: ID does not exist"
Jul 24 21:18:15.727929 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.727911    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:18:15.728225 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728210    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="kube-rbac-proxy"
Jul 24 21:18:15.728272 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728229    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="kube-rbac-proxy"
Jul 24 21:18:15.728272 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728237    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="alertmanager"
Jul 24 21:18:15.728272 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728244    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="alertmanager"
Jul 24 21:18:15.728272 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728253    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="prom-label-proxy"
Jul 24 21:18:15.728272 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728258    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="prom-label-proxy"
Jul 24 21:18:15.728272 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728264    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="thanos-sidecar"
Jul 24 21:18:15.728272 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728268    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="thanos-sidecar"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728276    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="prometheus"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728281    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="prometheus"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728290    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="init-config-reloader"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728298    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="init-config-reloader"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728304    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="kube-rbac-proxy"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728310    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="kube-rbac-proxy"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728318    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="config-reloader"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728322    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="config-reloader"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728330    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="kube-rbac-proxy-web"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728335    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="kube-rbac-proxy-web"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728344    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="init-config-reloader"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728349    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="init-config-reloader"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728355    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="kube-rbac-proxy-web"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728361    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="kube-rbac-proxy-web"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728371    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="kube-rbac-proxy-metric"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728379    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="kube-rbac-proxy-metric"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728384    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="config-reloader"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728389    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="config-reloader"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728394    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="kube-rbac-proxy-thanos"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728399    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="kube-rbac-proxy-thanos"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728454    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="thanos-sidecar"
Jul 24 21:18:15.728456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728463    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="kube-rbac-proxy-thanos"
Jul 24 21:18:15.729061 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728469    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="alertmanager"
Jul 24 21:18:15.729061 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728474    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="kube-rbac-proxy-web"
Jul 24 21:18:15.729061 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728481    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="kube-rbac-proxy-web"
Jul 24 21:18:15.729061 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728486    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="prometheus"
Jul 24 21:18:15.729061 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728492    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="kube-rbac-proxy"
Jul 24 21:18:15.729061 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728498    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" containerName="config-reloader"
Jul 24 21:18:15.729061 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728503    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="kube-rbac-proxy"
Jul 24 21:18:15.729061 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728507    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="kube-rbac-proxy-metric"
Jul 24 21:18:15.729061 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728511    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="config-reloader"
Jul 24 21:18:15.729061 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.728516    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7179b107-90f0-47da-991a-3f42a83d71ac" containerName="prom-label-proxy"
Jul 24 21:18:15.734226 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.734199    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.736800 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.736781    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 24 21:18:15.736932 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.736915    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 24 21:18:15.736999 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.736936    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 24 21:18:15.736999 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.736952    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 24 21:18:15.737426 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.737375    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 24 21:18:15.737426 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.737395    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 24 21:18:15.737426 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.737395    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 24 21:18:15.737600 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.737374    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5sv6q\""
Jul 24 21:18:15.737600 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.737400    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 24 21:18:15.737600 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.737374    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5h8ihnuhrvh18\""
Jul 24 21:18:15.737600 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.737562    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 24 21:18:15.737756 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.737744    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 24 21:18:15.737799 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.737777    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 24 21:18:15.737799 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.737788    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 24 21:18:15.741895 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.741873    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 24 21:18:15.743421 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.743406    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 24 21:18:15.754864 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.754833    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c888068-d60f-42eb-9fe1-5325a9f1f9a7" path="/var/lib/kubelet/pods/3c888068-d60f-42eb-9fe1-5325a9f1f9a7/volumes"
Jul 24 21:18:15.755363 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.755346    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:18:15.850355 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850325    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.850720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850362    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prvw7\" (UniqueName: \"kubernetes.io/projected/01482372-8206-4e25-943b-e98cf84a2c7b-kube-api-access-prvw7\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.850720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-config\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.850720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850444    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.850720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850489    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.850720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850520    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.850720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850545    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.850720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850578    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.850720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850636    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01482372-8206-4e25-943b-e98cf84a2c7b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.850720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850658    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.850720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850686    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.851197 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.851197 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01482372-8206-4e25-943b-e98cf84a2c7b-config-out\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.851197 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850801    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.851197 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.851197 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850842    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.851197 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850889    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.851197 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-web-config\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.851197 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01482372-8206-4e25-943b-e98cf84a2c7b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.851197 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.850959    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.951539 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951454    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-config\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.951539 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951503    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.951539 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951536    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.951753 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951555    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.951753 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951571    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.951753 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.951753 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951606    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01482372-8206-4e25-943b-e98cf84a2c7b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.951753 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951653    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.951753 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951682    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.951753 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951725    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.951753 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951753    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01482372-8206-4e25-943b-e98cf84a2c7b-config-out\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.952171 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951784    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.952171 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.952171 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951833    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.952171 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.952171 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-web-config\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.952171 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951921    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01482372-8206-4e25-943b-e98cf84a2c7b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.952171 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951947    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.952171 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.951977    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.952171 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.952002    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prvw7\" (UniqueName: \"kubernetes.io/projected/01482372-8206-4e25-943b-e98cf84a2c7b-kube-api-access-prvw7\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.952774 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.952289    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.952774 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.952661    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.954859 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.954830    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.956575 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.955308    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-config\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.956575 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.955407    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.956575 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.955421    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.956575 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.955466    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.956575 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.955510    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.956575 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.955615    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/01482372-8206-4e25-943b-e98cf84a2c7b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.956575 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.955807    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/01482372-8206-4e25-943b-e98cf84a2c7b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.956575 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.956407    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.956575 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.956455    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.956575 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.956531    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.957193 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.956628    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.957193 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.956800    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/01482372-8206-4e25-943b-e98cf84a2c7b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.957579 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.957560    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.957714 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.957696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/01482372-8206-4e25-943b-e98cf84a2c7b-config-out\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.958056 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.958032    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-web-config\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.958598 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.958579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/01482372-8206-4e25-943b-e98cf84a2c7b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:15.960457 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:15.960436    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prvw7\" (UniqueName: \"kubernetes.io/projected/01482372-8206-4e25-943b-e98cf84a2c7b-kube-api-access-prvw7\") pod \"prometheus-k8s-0\" (UID: \"01482372-8206-4e25-943b-e98cf84a2c7b\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:16.043673 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:16.043643    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:16.184847 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:16.184757    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 24 21:18:16.187886 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:18:16.187859    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01482372_8206_4e25_943b_e98cf84a2c7b.slice/crio-88d035697fd82083dfdb79feb15c0f4c3513de8970d71d5ff5262e231e3d0ac6 WatchSource:0}: Error finding container 88d035697fd82083dfdb79feb15c0f4c3513de8970d71d5ff5262e231e3d0ac6: Status 404 returned error can't find the container with id 88d035697fd82083dfdb79feb15c0f4c3513de8970d71d5ff5262e231e3d0ac6
Jul 24 21:18:16.665708 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:16.665675    2560 generic.go:356] "Generic (PLEG): container finished" podID="01482372-8206-4e25-943b-e98cf84a2c7b" containerID="38093cc70c8fa99e6995033b215b88f470379f9dc0e854b42afb374fcff3f203" exitCode=0
Jul 24 21:18:16.665944 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:16.665741    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01482372-8206-4e25-943b-e98cf84a2c7b","Type":"ContainerDied","Data":"38093cc70c8fa99e6995033b215b88f470379f9dc0e854b42afb374fcff3f203"}
Jul 24 21:18:16.665944 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:16.665791    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01482372-8206-4e25-943b-e98cf84a2c7b","Type":"ContainerStarted","Data":"88d035697fd82083dfdb79feb15c0f4c3513de8970d71d5ff5262e231e3d0ac6"}
Jul 24 21:18:17.673350 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:17.673316    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01482372-8206-4e25-943b-e98cf84a2c7b","Type":"ContainerStarted","Data":"af2e5c5343ed9b86f9af54db80600915fd5c04a0c94fc1f5a66f270c173f1160"}
Jul 24 21:18:17.673813 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:17.673408    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01482372-8206-4e25-943b-e98cf84a2c7b","Type":"ContainerStarted","Data":"8cd8752fdb6aadef6ee4598efdb81470b60f7f942d860093e9b10391a8d0a08f"}
Jul 24 21:18:17.673813 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:17.673434    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01482372-8206-4e25-943b-e98cf84a2c7b","Type":"ContainerStarted","Data":"976a8c01dea6b835f7f2f127bc12a29c418141f564e1144ce8e7dc6ca2de5b54"}
Jul 24 21:18:17.673813 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:17.673458    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01482372-8206-4e25-943b-e98cf84a2c7b","Type":"ContainerStarted","Data":"56c040fb60fbaf83e7dd7258a980c8f4141235aba3b731633e996528c9d1c335"}
Jul 24 21:18:17.673813 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:17.673481    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01482372-8206-4e25-943b-e98cf84a2c7b","Type":"ContainerStarted","Data":"a8ddf461c8ab254b36e81afd4d3b91a2b39ddcf4de6d18d9412b1cb110161310"}
Jul 24 21:18:17.673813 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:17.673506    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"01482372-8206-4e25-943b-e98cf84a2c7b","Type":"ContainerStarted","Data":"a500f3027583fbf641941b93b02bbf4f20a6495a7faae287ae2fa31140c1fe37"}
Jul 24 21:18:17.705267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:17.705199    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.68726264 podStartE2EDuration="2.705178661s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.916021ms" isStatefulPod=false podCreationTimestamp="2026-07-24 21:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:18:17.704173054 +0000 UTC m=+176.460518867" watchObservedRunningTime="2026-07-24 21:18:17.705178661 +0000 UTC m=+176.461524474"
Jul 24 21:18:19.580226 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:19.580193    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:19.649418 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:19.649382    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:21.044510 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:21.044469    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:18:30.910347 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:30.910313    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:31.011404 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:31.011371    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:18:31.077394 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:18:31.077365    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:06.389037 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:06.389006    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:06.771591 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:06.771515    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:07.631932 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:07.631900    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:16.044283 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:16.044249    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:19:16.060015 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:16.059991    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:19:16.903737 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:16.903711    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 24 21:19:24.670539 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:24.670506    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:24.772626 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:24.772588    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:27.247029 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:27.246984    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:36.671438 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:36.671400    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:36.772839 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:36.772808    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:39.826399 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:39.826365    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:39.926806 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:39.926773    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:41.731568 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:41.731534    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:41.828486 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:41.828379    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:55.295435 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:55.295404    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:55.362191 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:55.362159    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:55.853174 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:55.853142    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:19:55.918923 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:19:55.918894    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:06.864674 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:20:06.864646    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:06.964323 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:20:06.964287    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:07.047270 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:20:07.047240    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:21.612649 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:20:21.612621    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:20:21.616578 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:20:21.616556    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 21:20:21.625918 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:20:21.625896    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:20:23.731655 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:20:23.731618    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:23.842712 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:20:23.842684    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:23.937914 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:20:23.937883    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:43.739962 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:20:43.739929    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:51.732462 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:20:51.732403    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:51.829631 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:20:51.829599    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:52.123577 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:20:52.123544    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:52.221109 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:20:52.221075    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:20:52.324362 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:20:52.324334    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:14.841846 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:21:14.841811    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:14.951735 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:21:14.951701    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:21.170569 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:21:21.170531    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:21.789086 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:21:21.789051    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:21.971983 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:21:21.971957    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:22.569942 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:21:22.569911    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:22.776283 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:21:22.776243    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:22.907036 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:21:22.907002    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:23.015179 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:21:23.015149    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:23.255240 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:21:23.255152    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:23.691204 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:21:23.691176    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:23.918165 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:21:23.918110    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:24.092562 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:21:24.092466    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:21:24.198956 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:21:24.198919    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:42.170958 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:42.170877    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:42.273111 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:42.273085    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:46.882279 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:46.882242    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:46.980526 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:46.980486    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:47.040612 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.040582    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-664687df58-gx52s"]
Jul 24 21:22:47.043695 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.043680    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-gx52s"
Jul 24 21:22:47.051089 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.051062    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 24 21:22:47.053365 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.053349    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 24 21:22:47.054043 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.054028    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-47g52\""
Jul 24 21:22:47.059689 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.059673    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 24 21:22:47.103317 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.103290    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-664687df58-gx52s"]
Jul 24 21:22:47.140830 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.140761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r69wc\" (UniqueName: \"kubernetes.io/projected/311ac022-dc47-4381-9a63-1b2d4b06fab5-kube-api-access-r69wc\") pod \"kserve-controller-manager-664687df58-gx52s\" (UID: \"311ac022-dc47-4381-9a63-1b2d4b06fab5\") " pod="kserve/kserve-controller-manager-664687df58-gx52s"
Jul 24 21:22:47.140830 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.140809    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/311ac022-dc47-4381-9a63-1b2d4b06fab5-cert\") pod \"kserve-controller-manager-664687df58-gx52s\" (UID: \"311ac022-dc47-4381-9a63-1b2d4b06fab5\") " pod="kserve/kserve-controller-manager-664687df58-gx52s"
Jul 24 21:22:47.193995 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.193963    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:47.241470 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.241440    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r69wc\" (UniqueName: \"kubernetes.io/projected/311ac022-dc47-4381-9a63-1b2d4b06fab5-kube-api-access-r69wc\") pod \"kserve-controller-manager-664687df58-gx52s\" (UID: \"311ac022-dc47-4381-9a63-1b2d4b06fab5\") " pod="kserve/kserve-controller-manager-664687df58-gx52s"
Jul 24 21:22:47.241626 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.241480    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/311ac022-dc47-4381-9a63-1b2d4b06fab5-cert\") pod \"kserve-controller-manager-664687df58-gx52s\" (UID: \"311ac022-dc47-4381-9a63-1b2d4b06fab5\") " pod="kserve/kserve-controller-manager-664687df58-gx52s"
Jul 24 21:22:47.244095 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.244068    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/311ac022-dc47-4381-9a63-1b2d4b06fab5-cert\") pod \"kserve-controller-manager-664687df58-gx52s\" (UID: \"311ac022-dc47-4381-9a63-1b2d4b06fab5\") " pod="kserve/kserve-controller-manager-664687df58-gx52s"
Jul 24 21:22:47.271504 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.271478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r69wc\" (UniqueName: \"kubernetes.io/projected/311ac022-dc47-4381-9a63-1b2d4b06fab5-kube-api-access-r69wc\") pod \"kserve-controller-manager-664687df58-gx52s\" (UID: \"311ac022-dc47-4381-9a63-1b2d4b06fab5\") " pod="kserve/kserve-controller-manager-664687df58-gx52s"
Jul 24 21:22:47.353397 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.353363    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-gx52s"
Jul 24 21:22:47.379550 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.379525    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:47.484989 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.484953    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:47.515952 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.515932    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-664687df58-gx52s"]
Jul 24 21:22:47.518313 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:22:47.518287    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311ac022_dc47_4381_9a63_1b2d4b06fab5.slice/crio-f43413dc93b978ecffea654fcd409a3c59c612433201c0364845b82a08b785de WatchSource:0}: Error finding container f43413dc93b978ecffea654fcd409a3c59c612433201c0364845b82a08b785de: Status 404 returned error can't find the container with id f43413dc93b978ecffea654fcd409a3c59c612433201c0364845b82a08b785de
Jul 24 21:22:47.519663 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.519642    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 21:22:47.661779 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:47.661694    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-gx52s" event={"ID":"311ac022-dc47-4381-9a63-1b2d4b06fab5","Type":"ContainerStarted","Data":"f43413dc93b978ecffea654fcd409a3c59c612433201c0364845b82a08b785de"}
Jul 24 21:22:48.328887 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:48.328851    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:48.451269 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:48.451235    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:48.520346 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:48.520302    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:22:50.679803 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:50.679769    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-gx52s" event={"ID":"311ac022-dc47-4381-9a63-1b2d4b06fab5","Type":"ContainerStarted","Data":"704dddd39a0b943764e7ae90dbaf45cff1d0efc832b79dc3c46c42f2fe4622c6"}
Jul 24 21:22:50.680181 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:50.679895    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-664687df58-gx52s"
Jul 24 21:22:50.723893 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:22:50.723848    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-664687df58-gx52s" podStartSLOduration=2.211193393 podStartE2EDuration="4.723835283s" totalImagesPullingTime="2.51264189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:22:50.722711795 +0000 UTC m=+449.479057609" watchObservedRunningTime="2026-07-24 21:22:50.723835283 +0000 UTC m=+449.480181096"
Jul 24 21:23:07.516591 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:07.516557    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:07.586723 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:07.586689    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:10.060182 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:10.060149    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:10.140062 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:10.140033    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:21.688521 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:21.688492    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-664687df58-gx52s"
Jul 24 21:23:21.795241 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:21.795183    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:21.917713 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:21.917685    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:21.986551 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:21.986473    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:22.504268 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:22.504218    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:22.606119 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:22.606085    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:33.119698 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.119668    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:33.147549 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.147517    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-664687df58-gx52s"]
Jul 24 21:23:33.147739 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.147717    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-664687df58-gx52s" podUID="311ac022-dc47-4381-9a63-1b2d4b06fab5" containerName="manager" containerID="cri-o://704dddd39a0b943764e7ae90dbaf45cff1d0efc832b79dc3c46c42f2fe4622c6" gracePeriod=10
Jul 24 21:23:33.180622 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.180597    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-664687df58-pzmzn"]
Jul 24 21:23:33.182689 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.182675    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-pzmzn"
Jul 24 21:23:33.191951 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.191929    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-664687df58-pzmzn"]
Jul 24 21:23:33.209005 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.208979    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb930e16-8f61-4e32-ae81-37a49ab4f923-cert\") pod \"kserve-controller-manager-664687df58-pzmzn\" (UID: \"fb930e16-8f61-4e32-ae81-37a49ab4f923\") " pod="kserve/kserve-controller-manager-664687df58-pzmzn"
Jul 24 21:23:33.209149 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.209035    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lmc4\" (UniqueName: \"kubernetes.io/projected/fb930e16-8f61-4e32-ae81-37a49ab4f923-kube-api-access-2lmc4\") pod \"kserve-controller-manager-664687df58-pzmzn\" (UID: \"fb930e16-8f61-4e32-ae81-37a49ab4f923\") " pod="kserve/kserve-controller-manager-664687df58-pzmzn"
Jul 24 21:23:33.226073 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.226047    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:33.310512 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.310476    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb930e16-8f61-4e32-ae81-37a49ab4f923-cert\") pod \"kserve-controller-manager-664687df58-pzmzn\" (UID: \"fb930e16-8f61-4e32-ae81-37a49ab4f923\") " pod="kserve/kserve-controller-manager-664687df58-pzmzn"
Jul 24 21:23:33.310683 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.310538    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lmc4\" (UniqueName: \"kubernetes.io/projected/fb930e16-8f61-4e32-ae81-37a49ab4f923-kube-api-access-2lmc4\") pod \"kserve-controller-manager-664687df58-pzmzn\" (UID: \"fb930e16-8f61-4e32-ae81-37a49ab4f923\") " pod="kserve/kserve-controller-manager-664687df58-pzmzn"
Jul 24 21:23:33.312736 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.312715    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb930e16-8f61-4e32-ae81-37a49ab4f923-cert\") pod \"kserve-controller-manager-664687df58-pzmzn\" (UID: \"fb930e16-8f61-4e32-ae81-37a49ab4f923\") " pod="kserve/kserve-controller-manager-664687df58-pzmzn"
Jul 24 21:23:33.320583 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.320554    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lmc4\" (UniqueName: \"kubernetes.io/projected/fb930e16-8f61-4e32-ae81-37a49ab4f923-kube-api-access-2lmc4\") pod \"kserve-controller-manager-664687df58-pzmzn\" (UID: \"fb930e16-8f61-4e32-ae81-37a49ab4f923\") " pod="kserve/kserve-controller-manager-664687df58-pzmzn"
Jul 24 21:23:33.320727 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.320704    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:23:33.387704 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.387685    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-gx52s"
Jul 24 21:23:33.410982 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.410955    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/311ac022-dc47-4381-9a63-1b2d4b06fab5-cert\") pod \"311ac022-dc47-4381-9a63-1b2d4b06fab5\" (UID: \"311ac022-dc47-4381-9a63-1b2d4b06fab5\") "
Jul 24 21:23:33.411118 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.411019    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r69wc\" (UniqueName: \"kubernetes.io/projected/311ac022-dc47-4381-9a63-1b2d4b06fab5-kube-api-access-r69wc\") pod \"311ac022-dc47-4381-9a63-1b2d4b06fab5\" (UID: \"311ac022-dc47-4381-9a63-1b2d4b06fab5\") "
Jul 24 21:23:33.413419 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.413391    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311ac022-dc47-4381-9a63-1b2d4b06fab5-cert" (OuterVolumeSpecName: "cert") pod "311ac022-dc47-4381-9a63-1b2d4b06fab5" (UID: "311ac022-dc47-4381-9a63-1b2d4b06fab5"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:23:33.413514 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.413442    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311ac022-dc47-4381-9a63-1b2d4b06fab5-kube-api-access-r69wc" (OuterVolumeSpecName: "kube-api-access-r69wc") pod "311ac022-dc47-4381-9a63-1b2d4b06fab5" (UID: "311ac022-dc47-4381-9a63-1b2d4b06fab5"). InnerVolumeSpecName "kube-api-access-r69wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:23:33.512272 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.512243    2560 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/311ac022-dc47-4381-9a63-1b2d4b06fab5-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:23:33.512272 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.512268    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r69wc\" (UniqueName: \"kubernetes.io/projected/311ac022-dc47-4381-9a63-1b2d4b06fab5-kube-api-access-r69wc\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:23:33.529230 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.529205    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-pzmzn"
Jul 24 21:23:33.657696 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.657678    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-664687df58-pzmzn"]
Jul 24 21:23:33.660188 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:23:33.660164    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb930e16_8f61_4e32_ae81_37a49ab4f923.slice/crio-9b987ad14712be3d8a18ca4f0896c12a05bf2a14b72ce1357f087b70c90d6f8c WatchSource:0}: Error finding container 9b987ad14712be3d8a18ca4f0896c12a05bf2a14b72ce1357f087b70c90d6f8c: Status 404 returned error can't find the container with id 9b987ad14712be3d8a18ca4f0896c12a05bf2a14b72ce1357f087b70c90d6f8c
Jul 24 21:23:33.842007 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.841978    2560 generic.go:356] "Generic (PLEG): container finished" podID="311ac022-dc47-4381-9a63-1b2d4b06fab5" containerID="704dddd39a0b943764e7ae90dbaf45cff1d0efc832b79dc3c46c42f2fe4622c6" exitCode=0
Jul 24 21:23:33.842176 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.842030    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-664687df58-gx52s"
Jul 24 21:23:33.842176 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.842067    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-gx52s" event={"ID":"311ac022-dc47-4381-9a63-1b2d4b06fab5","Type":"ContainerDied","Data":"704dddd39a0b943764e7ae90dbaf45cff1d0efc832b79dc3c46c42f2fe4622c6"}
Jul 24 21:23:33.842176 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.842118    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-gx52s" event={"ID":"311ac022-dc47-4381-9a63-1b2d4b06fab5","Type":"ContainerDied","Data":"f43413dc93b978ecffea654fcd409a3c59c612433201c0364845b82a08b785de"}
Jul 24 21:23:33.842176 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.842167    2560 scope.go:117] "RemoveContainer" containerID="704dddd39a0b943764e7ae90dbaf45cff1d0efc832b79dc3c46c42f2fe4622c6"
Jul 24 21:23:33.843334 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.843305    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-pzmzn" event={"ID":"fb930e16-8f61-4e32-ae81-37a49ab4f923","Type":"ContainerStarted","Data":"9b987ad14712be3d8a18ca4f0896c12a05bf2a14b72ce1357f087b70c90d6f8c"}
Jul 24 21:23:33.850062 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.850047    2560 scope.go:117] "RemoveContainer" containerID="704dddd39a0b943764e7ae90dbaf45cff1d0efc832b79dc3c46c42f2fe4622c6"
Jul 24 21:23:33.850338 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:23:33.850315    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704dddd39a0b943764e7ae90dbaf45cff1d0efc832b79dc3c46c42f2fe4622c6\": container with ID starting with 704dddd39a0b943764e7ae90dbaf45cff1d0efc832b79dc3c46c42f2fe4622c6 not found: ID does not exist" containerID="704dddd39a0b943764e7ae90dbaf45cff1d0efc832b79dc3c46c42f2fe4622c6"
Jul 24 21:23:33.850415 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.850346    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704dddd39a0b943764e7ae90dbaf45cff1d0efc832b79dc3c46c42f2fe4622c6"} err="failed to get container status \"704dddd39a0b943764e7ae90dbaf45cff1d0efc832b79dc3c46c42f2fe4622c6\": rpc error: code = NotFound desc = could not find container \"704dddd39a0b943764e7ae90dbaf45cff1d0efc832b79dc3c46c42f2fe4622c6\": container with ID starting with 704dddd39a0b943764e7ae90dbaf45cff1d0efc832b79dc3c46c42f2fe4622c6 not found: ID does not exist"
Jul 24 21:23:33.858892 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.858873    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-664687df58-gx52s"]
Jul 24 21:23:33.863913 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:33.863896    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-664687df58-gx52s"]
Jul 24 21:23:34.848524 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:34.848488    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-664687df58-pzmzn" event={"ID":"fb930e16-8f61-4e32-ae81-37a49ab4f923","Type":"ContainerStarted","Data":"abced786514d6540902984664c9988c1e070a9d3aed3286ea475c44a49f17113"}
Jul 24 21:23:34.848968 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:34.848573    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-664687df58-pzmzn"
Jul 24 21:23:34.866497 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:34.866424    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-664687df58-pzmzn" podStartSLOduration=1.4639682569999999 podStartE2EDuration="1.866407541s" totalImagesPullingTime="402.439284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:23:34.865280209 +0000 UTC m=+493.621626022" watchObservedRunningTime="2026-07-24 21:23:34.866407541 +0000 UTC m=+493.622753360"
Jul 24 21:23:35.753535 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:23:35.753501    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="311ac022-dc47-4381-9a63-1b2d4b06fab5" path="/var/lib/kubelet/pods/311ac022-dc47-4381-9a63-1b2d4b06fab5/volumes"
Jul 24 21:24:05.857778 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:05.857743    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-664687df58-pzmzn"
Jul 24 21:24:06.985113 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:06.985067    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:07.086292 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:07.086257    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:07.198326 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:07.198283    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:07.300615 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:07.300532    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:08.071621 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:08.071583    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:08.145815 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:08.145790    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:22.496932 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.496896    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-pm56h"]
Jul 24 21:24:22.497321 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.497242    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311ac022-dc47-4381-9a63-1b2d4b06fab5" containerName="manager"
Jul 24 21:24:22.497321 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.497256    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="311ac022-dc47-4381-9a63-1b2d4b06fab5" containerName="manager"
Jul 24 21:24:22.497321 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.497319    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="311ac022-dc47-4381-9a63-1b2d4b06fab5" containerName="manager"
Jul 24 21:24:22.501539 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.501520    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pm56h"
Jul 24 21:24:22.504251 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.504232    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 24 21:24:22.504365 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.504279    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-c92nt\""
Jul 24 21:24:22.510317 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.510296    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pm56h"]
Jul 24 21:24:22.607508 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.607474    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/906bf867-5ad4-4a5c-85fc-0724afe0f329-model-cache\") pod \"s3-init-pm56h\" (UID: \"906bf867-5ad4-4a5c-85fc-0724afe0f329\") " pod="kserve/s3-init-pm56h"
Jul 24 21:24:22.607665 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.607516    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwrw4\" (UniqueName: \"kubernetes.io/projected/906bf867-5ad4-4a5c-85fc-0724afe0f329-kube-api-access-lwrw4\") pod \"s3-init-pm56h\" (UID: \"906bf867-5ad4-4a5c-85fc-0724afe0f329\") " pod="kserve/s3-init-pm56h"
Jul 24 21:24:22.708590 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.708556    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/906bf867-5ad4-4a5c-85fc-0724afe0f329-model-cache\") pod \"s3-init-pm56h\" (UID: \"906bf867-5ad4-4a5c-85fc-0724afe0f329\") " pod="kserve/s3-init-pm56h"
Jul 24 21:24:22.708777 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.708612    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwrw4\" (UniqueName: \"kubernetes.io/projected/906bf867-5ad4-4a5c-85fc-0724afe0f329-kube-api-access-lwrw4\") pod \"s3-init-pm56h\" (UID: \"906bf867-5ad4-4a5c-85fc-0724afe0f329\") " pod="kserve/s3-init-pm56h"
Jul 24 21:24:22.708952 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.708933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/906bf867-5ad4-4a5c-85fc-0724afe0f329-model-cache\") pod \"s3-init-pm56h\" (UID: \"906bf867-5ad4-4a5c-85fc-0724afe0f329\") " pod="kserve/s3-init-pm56h"
Jul 24 21:24:22.717827 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.717794    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwrw4\" (UniqueName: \"kubernetes.io/projected/906bf867-5ad4-4a5c-85fc-0724afe0f329-kube-api-access-lwrw4\") pod \"s3-init-pm56h\" (UID: \"906bf867-5ad4-4a5c-85fc-0724afe0f329\") " pod="kserve/s3-init-pm56h"
Jul 24 21:24:22.823900 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.823809    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pm56h"
Jul 24 21:24:22.956759 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:22.956731    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pm56h"]
Jul 24 21:24:22.959847 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:24:22.959810    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod906bf867_5ad4_4a5c_85fc_0724afe0f329.slice/crio-6eb2fc9bf0985d865eb84cd20eb832c786dfe4f1b16e294f4b7dee0383562321 WatchSource:0}: Error finding container 6eb2fc9bf0985d865eb84cd20eb832c786dfe4f1b16e294f4b7dee0383562321: Status 404 returned error can't find the container with id 6eb2fc9bf0985d865eb84cd20eb832c786dfe4f1b16e294f4b7dee0383562321
Jul 24 21:24:23.041884 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:23.041851    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pm56h" event={"ID":"906bf867-5ad4-4a5c-85fc-0724afe0f329","Type":"ContainerStarted","Data":"6eb2fc9bf0985d865eb84cd20eb832c786dfe4f1b16e294f4b7dee0383562321"}
Jul 24 21:24:25.642193 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:25.642161    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:25.706080 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:25.706048    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:24:26.056024 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:26.055939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pm56h" event={"ID":"906bf867-5ad4-4a5c-85fc-0724afe0f329","Type":"ContainerStarted","Data":"9734d4395489a90bf888e23d2020f03b44312d64406f8bbcb4f001bddaf8086f"}
Jul 24 21:24:28.069375 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:28.069293    2560 generic.go:356] "Generic (PLEG): container finished" podID="906bf867-5ad4-4a5c-85fc-0724afe0f329" containerID="9734d4395489a90bf888e23d2020f03b44312d64406f8bbcb4f001bddaf8086f" exitCode=0
Jul 24 21:24:28.069375 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:28.069366    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pm56h" event={"ID":"906bf867-5ad4-4a5c-85fc-0724afe0f329","Type":"ContainerDied","Data":"9734d4395489a90bf888e23d2020f03b44312d64406f8bbcb4f001bddaf8086f"}
Jul 24 21:24:33.094916 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:33.094879    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pm56h" event={"ID":"906bf867-5ad4-4a5c-85fc-0724afe0f329","Type":"ContainerStarted","Data":"36120b70e55ed054588abca2b230922ad47cfbb0a652abd158b10161cf08d77e"}
Jul 24 21:24:33.110159 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:24:33.110099    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-pm56h" podStartSLOduration=1.937992763 podStartE2EDuration="11.110083602s" totalImagesPullingTime="7.202079479s" totalInitContainerRuntime="1.97001136s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:24:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:24:33.109261406 +0000 UTC m=+551.865607223" watchObservedRunningTime="2026-07-24 21:24:33.110083602 +0000 UTC m=+551.866429414"
Jul 24 21:25:21.639558 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:21.639521    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:25:21.648794 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:21.648773    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:25:38.346388 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:38.346354    2560 generic.go:356] "Generic (PLEG): container finished" podID="906bf867-5ad4-4a5c-85fc-0724afe0f329" containerID="36120b70e55ed054588abca2b230922ad47cfbb0a652abd158b10161cf08d77e" exitCode=0
Jul 24 21:25:38.346797 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:38.346410    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pm56h" event={"ID":"906bf867-5ad4-4a5c-85fc-0724afe0f329","Type":"ContainerDied","Data":"36120b70e55ed054588abca2b230922ad47cfbb0a652abd158b10161cf08d77e"}
Jul 24 21:25:39.473613 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:39.473589    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pm56h"
Jul 24 21:25:39.533933 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:39.533905    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/906bf867-5ad4-4a5c-85fc-0724afe0f329-model-cache\") pod \"906bf867-5ad4-4a5c-85fc-0724afe0f329\" (UID: \"906bf867-5ad4-4a5c-85fc-0724afe0f329\") "
Jul 24 21:25:39.534099 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:39.533961    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwrw4\" (UniqueName: \"kubernetes.io/projected/906bf867-5ad4-4a5c-85fc-0724afe0f329-kube-api-access-lwrw4\") pod \"906bf867-5ad4-4a5c-85fc-0724afe0f329\" (UID: \"906bf867-5ad4-4a5c-85fc-0724afe0f329\") "
Jul 24 21:25:39.536105 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:39.536082    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906bf867-5ad4-4a5c-85fc-0724afe0f329-kube-api-access-lwrw4" (OuterVolumeSpecName: "kube-api-access-lwrw4") pod "906bf867-5ad4-4a5c-85fc-0724afe0f329" (UID: "906bf867-5ad4-4a5c-85fc-0724afe0f329"). InnerVolumeSpecName "kube-api-access-lwrw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:25:39.552593 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:39.552553    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/906bf867-5ad4-4a5c-85fc-0724afe0f329-model-cache" (OuterVolumeSpecName: "model-cache") pod "906bf867-5ad4-4a5c-85fc-0724afe0f329" (UID: "906bf867-5ad4-4a5c-85fc-0724afe0f329"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:25:39.634807 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:39.634778    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwrw4\" (UniqueName: \"kubernetes.io/projected/906bf867-5ad4-4a5c-85fc-0724afe0f329-kube-api-access-lwrw4\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:25:39.634807 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:39.634803    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/906bf867-5ad4-4a5c-85fc-0724afe0f329-model-cache\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:25:40.356099 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:40.356073    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pm56h"
Jul 24 21:25:40.356298 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:40.356066    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pm56h" event={"ID":"906bf867-5ad4-4a5c-85fc-0724afe0f329","Type":"ContainerDied","Data":"6eb2fc9bf0985d865eb84cd20eb832c786dfe4f1b16e294f4b7dee0383562321"}
Jul 24 21:25:40.356298 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:40.356196    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eb2fc9bf0985d865eb84cd20eb832c786dfe4f1b16e294f4b7dee0383562321"
Jul 24 21:25:40.742411 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:40.742377    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:25:40.842823 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:40.842793    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:25:52.404184 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.404100    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:25:52.466261 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.466224    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"]
Jul 24 21:25:52.466521 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.466509    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906bf867-5ad4-4a5c-85fc-0724afe0f329" containerName="s3-init"
Jul 24 21:25:52.466521 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.466522    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="906bf867-5ad4-4a5c-85fc-0724afe0f329" containerName="s3-init"
Jul 24 21:25:52.466630 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.466552    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906bf867-5ad4-4a5c-85fc-0724afe0f329" containerName="download-hf-model"
Jul 24 21:25:52.466630 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.466557    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="906bf867-5ad4-4a5c-85fc-0724afe0f329" containerName="download-hf-model"
Jul 24 21:25:52.466630 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.466620    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="906bf867-5ad4-4a5c-85fc-0724afe0f329" containerName="s3-init"
Jul 24 21:25:52.469753 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.469739    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.472285 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.472247    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:25:52.472285 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.472279    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:25:52.472447 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.472332    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:25:52.472447 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.472338    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-05405-kube-rbac-proxy-sar-config\""
Jul 24 21:25:52.472968 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.472953    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-05405-predictor-serving-cert\""
Jul 24 21:25:52.472968 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.472962    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dm6jh\""
Jul 24 21:25:52.473070 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.472964    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:25:52.479268 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.479249    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"]
Jul 24 21:25:52.540474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.540443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b760f7-9174-4731-a696-219ae8b18a4d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.540636 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.540486    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b760f7-9174-4731-a696-219ae8b18a4d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.540636 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.540521    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71b760f7-9174-4731-a696-219ae8b18a4d-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.540636 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.540556    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71b760f7-9174-4731-a696-219ae8b18a4d-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.540636 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.540591    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xlrx\" (UniqueName: \"kubernetes.io/projected/71b760f7-9174-4731-a696-219ae8b18a4d-kube-api-access-5xlrx\") pod \"isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.641892 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.641855    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71b760f7-9174-4731-a696-219ae8b18a4d-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.641892 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.641897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xlrx\" (UniqueName: \"kubernetes.io/projected/71b760f7-9174-4731-a696-219ae8b18a4d-kube-api-access-5xlrx\") pod \"isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.642244 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.641937    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b760f7-9174-4731-a696-219ae8b18a4d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.642244 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.641959    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b760f7-9174-4731-a696-219ae8b18a4d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.642244 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.641984    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71b760f7-9174-4731-a696-219ae8b18a4d-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.642468 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.642444    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b760f7-9174-4731-a696-219ae8b18a4d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.642663 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.642640    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71b760f7-9174-4731-a696-219ae8b18a4d-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.642904 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.642666    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71b760f7-9174-4731-a696-219ae8b18a4d-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.644534 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.644515    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b760f7-9174-4731-a696-219ae8b18a4d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.651688 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.651669    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xlrx\" (UniqueName: \"kubernetes.io/projected/71b760f7-9174-4731-a696-219ae8b18a4d-kube-api-access-5xlrx\") pod \"isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.781220 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.781118    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:25:52.909613 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:52.909588    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"]
Jul 24 21:25:52.912200 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:25:52.912171    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b760f7_9174_4731_a696_219ae8b18a4d.slice/crio-403123d412972125c665a221e9e9372b2ff646e7c9577c4e24887db5a1c524a3 WatchSource:0}: Error finding container 403123d412972125c665a221e9e9372b2ff646e7c9577c4e24887db5a1c524a3: Status 404 returned error can't find the container with id 403123d412972125c665a221e9e9372b2ff646e7c9577c4e24887db5a1c524a3
Jul 24 21:25:53.408978 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:53.408943    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" event={"ID":"71b760f7-9174-4731-a696-219ae8b18a4d","Type":"ContainerStarted","Data":"01cb436745607abc5eb1f16ab83dbb88ced4e90c1dc93c6f237b631a8b0278b6"}
Jul 24 21:25:53.409368 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:53.409007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" event={"ID":"71b760f7-9174-4731-a696-219ae8b18a4d","Type":"ContainerStarted","Data":"403123d412972125c665a221e9e9372b2ff646e7c9577c4e24887db5a1c524a3"}
Jul 24 21:25:53.556758 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:53.556723    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:25:55.837666 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:55.837628    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:25:57.428321 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:57.428287    2560 generic.go:356] "Generic (PLEG): container finished" podID="71b760f7-9174-4731-a696-219ae8b18a4d" containerID="01cb436745607abc5eb1f16ab83dbb88ced4e90c1dc93c6f237b631a8b0278b6" exitCode=0
Jul 24 21:25:57.428321 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:25:57.428323    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" event={"ID":"71b760f7-9174-4731-a696-219ae8b18a4d","Type":"ContainerDied","Data":"01cb436745607abc5eb1f16ab83dbb88ced4e90c1dc93c6f237b631a8b0278b6"}
Jul 24 21:26:07.713910 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:07.713852    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:26:11.506147 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:11.506100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" event={"ID":"71b760f7-9174-4731-a696-219ae8b18a4d","Type":"ContainerStarted","Data":"33c14045254bd95762b8518d1ca1aa5c8dc7f6f3bc7caa36b60beffc186f5bd8"}
Jul 24 21:26:13.515671 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:13.515581    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" event={"ID":"71b760f7-9174-4731-a696-219ae8b18a4d","Type":"ContainerStarted","Data":"fc9793ff90e6c532f469b922391db37b532bcb744606882f1f67ca231b046308"}
Jul 24 21:26:16.530297 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:16.530255    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" event={"ID":"71b760f7-9174-4731-a696-219ae8b18a4d","Type":"ContainerStarted","Data":"a66b5d224748cbac0b011bb5f9169cb1fdc7b157c2503d62d130d2f91ad729df"}
Jul 24 21:26:16.530726 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:16.530487    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:26:16.530726 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:16.530556    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:26:16.530726 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:16.530585    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:26:16.532004 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:16.531960    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 24 21:26:16.532593 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:16.532572    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:26:16.552163 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:16.552111    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podStartSLOduration=2.333823166 podStartE2EDuration="24.55209402s" totalImagesPullingTime="18.400043128s" totalInitContainerRuntime="3.818227726s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:25:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:26:16.550109634 +0000 UTC m=+655.306455447" watchObservedRunningTime="2026-07-24 21:26:16.55209402 +0000 UTC m=+655.308439846"
Jul 24 21:26:17.534046 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:17.533999    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 24 21:26:17.534534 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:17.534512    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:26:22.538195 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:22.538166    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:26:22.538775 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:22.538747    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 24 21:26:22.539000 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:22.538979    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:26:32.539265 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:32.539216    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 24 21:26:32.539744 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:32.539700    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:26:42.539264 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:42.539205    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 24 21:26:42.539744 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:42.539711    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:26:52.539434 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:52.539392    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 24 21:26:52.539904 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:26:52.539883    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:02.538729 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:02.538678    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 24 21:27:02.539291 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:02.539144    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:03.430256 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:03.430226    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:12.539402 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:12.539360    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 24 21:27:12.539880 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:12.539806    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:22.538846 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:22.538796    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 24 21:27:22.539323 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:22.539297    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:32.539301 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:32.539269    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:27:32.539775 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:32.539543    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:27:47.663445 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.663415    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:47.736351 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.736320    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"]
Jul 24 21:27:47.747663 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.747629    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"]
Jul 24 21:27:47.747816 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.747749    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:47.747889 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.747844    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container" containerID="cri-o://33c14045254bd95762b8518d1ca1aa5c8dc7f6f3bc7caa36b60beffc186f5bd8" gracePeriod=30
Jul 24 21:27:47.747949 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.747904    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kube-rbac-proxy" containerID="cri-o://fc9793ff90e6c532f469b922391db37b532bcb744606882f1f67ca231b046308" gracePeriod=30
Jul 24 21:27:47.747949 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.747920    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent" containerID="cri-o://a66b5d224748cbac0b011bb5f9169cb1fdc7b157c2503d62d130d2f91ad729df" gracePeriod=30
Jul 24 21:27:47.750445 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.750236    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-8100c-predictor-serving-cert\""
Jul 24 21:27:47.750445 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.750243    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-8100c-kube-rbac-proxy-sar-config\""
Jul 24 21:27:47.755601 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.755422    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"]
Jul 24 21:27:47.759481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.759459    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:47.866537 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.866508    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:47.870983 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.870943    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"]
Jul 24 21:27:47.874452 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.874433    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:47.876614 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.876590    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-8100c-predictor-serving-cert\""
Jul 24 21:27:47.877142 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.877106    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-8100c-kube-rbac-proxy-sar-config\""
Jul 24 21:27:47.887839 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.887773    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"]
Jul 24 21:27:47.890299 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.890277    2560 generic.go:356] "Generic (PLEG): container finished" podID="71b760f7-9174-4731-a696-219ae8b18a4d" containerID="fc9793ff90e6c532f469b922391db37b532bcb744606882f1f67ca231b046308" exitCode=2
Jul 24 21:27:47.890401 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.890358    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" event={"ID":"71b760f7-9174-4731-a696-219ae8b18a4d","Type":"ContainerDied","Data":"fc9793ff90e6c532f469b922391db37b532bcb744606882f1f67ca231b046308"}
Jul 24 21:27:47.916691 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.916632    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0469501-f68e-4b73-99cf-4c0e36fbb284-cabundle-cert\") pod \"isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:47.916691 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.916668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzb5x\" (UniqueName: \"kubernetes.io/projected/c0469501-f68e-4b73-99cf-4c0e36fbb284-kube-api-access-dzb5x\") pod \"isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:47.916822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.916757    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0469501-f68e-4b73-99cf-4c0e36fbb284-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:47.916822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.916793    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0469501-f68e-4b73-99cf-4c0e36fbb284-proxy-tls\") pod \"isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:47.916822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.916814    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0469501-f68e-4b73-99cf-4c0e36fbb284-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:47.930627 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:47.930608    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:48.017822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.017792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0469501-f68e-4b73-99cf-4c0e36fbb284-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:48.017822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.017823    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0469501-f68e-4b73-99cf-4c0e36fbb284-proxy-tls\") pod \"isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:48.017999 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.017844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.017999 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.017942    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0469501-f68e-4b73-99cf-4c0e36fbb284-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:48.017999 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.017979    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.018167 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.018003    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6059f7-c136-4628-bc34-6403bbfe0eaf-proxy-tls\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.018167 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.018060    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0469501-f68e-4b73-99cf-4c0e36fbb284-cabundle-cert\") pod \"isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:48.018167 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.018113    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzb5x\" (UniqueName: \"kubernetes.io/projected/c0469501-f68e-4b73-99cf-4c0e36fbb284-kube-api-access-dzb5x\") pod \"isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:48.018385 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.018179    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c6059f7-c136-4628-bc34-6403bbfe0eaf-cabundle-cert\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.018385 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.018228    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7d89\" (UniqueName: \"kubernetes.io/projected/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kube-api-access-g7d89\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.018385 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.018252    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0469501-f68e-4b73-99cf-4c0e36fbb284-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:48.018627 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.018608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0469501-f68e-4b73-99cf-4c0e36fbb284-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:48.018682 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.018667    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0469501-f68e-4b73-99cf-4c0e36fbb284-cabundle-cert\") pod \"isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:48.020510 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.020493    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0469501-f68e-4b73-99cf-4c0e36fbb284-proxy-tls\") pod \"isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:48.026264 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.026243    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzb5x\" (UniqueName: \"kubernetes.io/projected/c0469501-f68e-4b73-99cf-4c0e36fbb284-kube-api-access-dzb5x\") pod \"isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:48.069330 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.069299    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:48.118971 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.118931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c6059f7-c136-4628-bc34-6403bbfe0eaf-cabundle-cert\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.119113 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.118999    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7d89\" (UniqueName: \"kubernetes.io/projected/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kube-api-access-g7d89\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.119113 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.119052    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.119113 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.119101    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.119309 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.119165    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6059f7-c136-4628-bc34-6403bbfe0eaf-proxy-tls\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.119717 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:27:48.119315    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-serving-cert: secret "isvc-xgboost-graph-raw-8100c-predictor-serving-cert" not found
Jul 24 21:27:48.119717 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:27:48.119379    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c6059f7-c136-4628-bc34-6403bbfe0eaf-proxy-tls podName:4c6059f7-c136-4628-bc34-6403bbfe0eaf nodeName:}" failed. No retries permitted until 2026-07-24 21:27:48.619359422 +0000 UTC m=+747.375705218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4c6059f7-c136-4628-bc34-6403bbfe0eaf-proxy-tls") pod "isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" (UID: "4c6059f7-c136-4628-bc34-6403bbfe0eaf") : secret "isvc-xgboost-graph-raw-8100c-predictor-serving-cert" not found
Jul 24 21:27:48.119717 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.119661    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c6059f7-c136-4628-bc34-6403bbfe0eaf-cabundle-cert\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.120045 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.119937    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.120494 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.120471    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.129106 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.129068    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7d89\" (UniqueName: \"kubernetes.io/projected/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kube-api-access-g7d89\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.207234 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.207140    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"]
Jul 24 21:27:48.209864 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:27:48.209841    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0469501_f68e_4b73_99cf_4c0e36fbb284.slice/crio-3767f956f23796cd2b4275a6b873858e039f27725ebbfb4d424016fa0a5ef475 WatchSource:0}: Error finding container 3767f956f23796cd2b4275a6b873858e039f27725ebbfb4d424016fa0a5ef475: Status 404 returned error can't find the container with id 3767f956f23796cd2b4275a6b873858e039f27725ebbfb4d424016fa0a5ef475
Jul 24 21:27:48.292902 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.292876    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:27:48.624537 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.624508    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6059f7-c136-4628-bc34-6403bbfe0eaf-proxy-tls\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.626910 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.626885    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6059f7-c136-4628-bc34-6403bbfe0eaf-proxy-tls\") pod \"isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.786902 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.786868    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:27:48.897846 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.897765    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" event={"ID":"c0469501-f68e-4b73-99cf-4c0e36fbb284","Type":"ContainerStarted","Data":"8b4a51da16ff691adb2e46c2fbc42aacca0868b589de970681286e3256b3b384"}
Jul 24 21:27:48.897846 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.897840    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" event={"ID":"c0469501-f68e-4b73-99cf-4c0e36fbb284","Type":"ContainerStarted","Data":"3767f956f23796cd2b4275a6b873858e039f27725ebbfb4d424016fa0a5ef475"}
Jul 24 21:27:48.921102 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:48.921060    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"]
Jul 24 21:27:48.923249 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:27:48.923226    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6059f7_c136_4628_bc34_6403bbfe0eaf.slice/crio-dda55dee947584e00d52beb7a1ff0e94997c46f60befcd8bd9f01ef676bf0cbd WatchSource:0}: Error finding container dda55dee947584e00d52beb7a1ff0e94997c46f60befcd8bd9f01ef676bf0cbd: Status 404 returned error can't find the container with id dda55dee947584e00d52beb7a1ff0e94997c46f60befcd8bd9f01ef676bf0cbd
Jul 24 21:27:49.903356 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:49.903319    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" event={"ID":"4c6059f7-c136-4628-bc34-6403bbfe0eaf","Type":"ContainerStarted","Data":"434a11f4d70d5a2acef884a3d1b518d506a7fd8df8b36219e5467953fb1c4e1c"}
Jul 24 21:27:49.903781 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:49.903391    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" event={"ID":"4c6059f7-c136-4628-bc34-6403bbfe0eaf","Type":"ContainerStarted","Data":"dda55dee947584e00d52beb7a1ff0e94997c46f60befcd8bd9f01ef676bf0cbd"}
Jul 24 21:27:52.534482 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:52.534406    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 24 21:27:52.538865 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:52.538841    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 24 21:27:52.539212 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:52.539181    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:27:52.917781 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:52.917751    2560 generic.go:356] "Generic (PLEG): container finished" podID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerID="8b4a51da16ff691adb2e46c2fbc42aacca0868b589de970681286e3256b3b384" exitCode=0
Jul 24 21:27:52.917940 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:52.917816    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" event={"ID":"c0469501-f68e-4b73-99cf-4c0e36fbb284","Type":"ContainerDied","Data":"8b4a51da16ff691adb2e46c2fbc42aacca0868b589de970681286e3256b3b384"}
Jul 24 21:27:52.920320 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:52.920296    2560 generic.go:356] "Generic (PLEG): container finished" podID="71b760f7-9174-4731-a696-219ae8b18a4d" containerID="33c14045254bd95762b8518d1ca1aa5c8dc7f6f3bc7caa36b60beffc186f5bd8" exitCode=0
Jul 24 21:27:52.920450 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:52.920361    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" event={"ID":"71b760f7-9174-4731-a696-219ae8b18a4d","Type":"ContainerDied","Data":"33c14045254bd95762b8518d1ca1aa5c8dc7f6f3bc7caa36b60beffc186f5bd8"}
Jul 24 21:27:52.921906 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:52.921886    2560 generic.go:356] "Generic (PLEG): container finished" podID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerID="434a11f4d70d5a2acef884a3d1b518d506a7fd8df8b36219e5467953fb1c4e1c" exitCode=0
Jul 24 21:27:52.922016 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:52.921946    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" event={"ID":"4c6059f7-c136-4628-bc34-6403bbfe0eaf","Type":"ContainerDied","Data":"434a11f4d70d5a2acef884a3d1b518d506a7fd8df8b36219e5467953fb1c4e1c"}
Jul 24 21:27:52.923187 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:52.923172    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 21:27:53.931290 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:53.931250    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" event={"ID":"c0469501-f68e-4b73-99cf-4c0e36fbb284","Type":"ContainerStarted","Data":"e4b22766f5210e702652754405ea210ba0d6d58b2d4753e8a2bf76bd46c2ce0b"}
Jul 24 21:27:53.931711 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:53.931341    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" event={"ID":"c0469501-f68e-4b73-99cf-4c0e36fbb284","Type":"ContainerStarted","Data":"c77ea050e6a58d4aef5503dcceb5e07bfd220c88d6c47747e5af8d8955a7039f"}
Jul 24 21:27:53.931711 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:53.931650    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:53.951921 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:53.951869    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" podStartSLOduration=2.969870745 podStartE2EDuration="6.951850524s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.981979779s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:27:53.949742908 +0000 UTC m=+752.706088736" watchObservedRunningTime="2026-07-24 21:27:53.951850524 +0000 UTC m=+752.708196407"
Jul 24 21:27:54.938484 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:54.938366    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:27:54.939482 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:54.939445    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 24 21:27:55.944746 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:55.944697    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 24 21:27:57.534995 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:27:57.534949    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 24 21:28:00.949846 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:00.949817    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:28:00.950513 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:00.950481    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 24 21:28:02.534908 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:02.534869    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 24 21:28:02.535310 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:02.535005    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:28:02.538634 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:02.538596    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 24 21:28:02.538953 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:02.538932    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:28:07.534304 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:07.534255    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 24 21:28:10.950668 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:10.950629    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 24 21:28:12.024598 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:12.024561    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" event={"ID":"4c6059f7-c136-4628-bc34-6403bbfe0eaf","Type":"ContainerStarted","Data":"9f432ac84c600f47e6929833b31de314d3de6a5631f8b1203df0c5eb79c729b3"}
Jul 24 21:28:12.024958 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:12.024622    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" event={"ID":"4c6059f7-c136-4628-bc34-6403bbfe0eaf","Type":"ContainerStarted","Data":"47e9ded3a5efc8a381b467213ed40f14d024dfc624bb6fb5f04bf1da54915df3"}
Jul 24 21:28:12.024998 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:12.024953    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:28:12.025104 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:12.025087    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:28:12.026400 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:12.026372    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 24 21:28:12.044090 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:12.043740    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" podStartSLOduration=3.080392123 podStartE2EDuration="25.043725165s" totalImagesPullingTime="18.092471586s" totalInitContainerRuntime="3.870861456s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:28:12.043379109 +0000 UTC m=+770.799724923" watchObservedRunningTime="2026-07-24 21:28:12.043725165 +0000 UTC m=+770.800070979"
Jul 24 21:28:12.534867 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:12.534826    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 24 21:28:12.539246 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:12.539215    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 24 21:28:12.539360 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:12.539343    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:28:12.539574 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:12.539554    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:28:12.539658 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:12.539646    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:28:13.028631 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:13.028593    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 24 21:28:14.033162 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:14.033096    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 24 21:28:17.534833 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:17.534790    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 24 21:28:18.052743 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.052710    2560 generic.go:356] "Generic (PLEG): container finished" podID="71b760f7-9174-4731-a696-219ae8b18a4d" containerID="a66b5d224748cbac0b011bb5f9169cb1fdc7b157c2503d62d130d2f91ad729df" exitCode=0
Jul 24 21:28:18.052896 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.052772    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" event={"ID":"71b760f7-9174-4731-a696-219ae8b18a4d","Type":"ContainerDied","Data":"a66b5d224748cbac0b011bb5f9169cb1fdc7b157c2503d62d130d2f91ad729df"}
Jul 24 21:28:18.431500 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.431474    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:28:18.454757 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.454730    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b760f7-9174-4731-a696-219ae8b18a4d-proxy-tls\") pod \"71b760f7-9174-4731-a696-219ae8b18a4d\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") "
Jul 24 21:28:18.454926 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.454775    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71b760f7-9174-4731-a696-219ae8b18a4d-kube-rbac-proxy-sar-config\") pod \"71b760f7-9174-4731-a696-219ae8b18a4d\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") "
Jul 24 21:28:18.454926 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.454844    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71b760f7-9174-4731-a696-219ae8b18a4d-cabundle-cert\") pod \"71b760f7-9174-4731-a696-219ae8b18a4d\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") "
Jul 24 21:28:18.454926 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.454871    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xlrx\" (UniqueName: \"kubernetes.io/projected/71b760f7-9174-4731-a696-219ae8b18a4d-kube-api-access-5xlrx\") pod \"71b760f7-9174-4731-a696-219ae8b18a4d\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") "
Jul 24 21:28:18.456037 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.455348    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b760f7-9174-4731-a696-219ae8b18a4d-kserve-provision-location\") pod \"71b760f7-9174-4731-a696-219ae8b18a4d\" (UID: \"71b760f7-9174-4731-a696-219ae8b18a4d\") "
Jul 24 21:28:18.456037 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.455554    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b760f7-9174-4731-a696-219ae8b18a4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "71b760f7-9174-4731-a696-219ae8b18a4d" (UID: "71b760f7-9174-4731-a696-219ae8b18a4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:28:18.456037 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.455674    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71b760f7-9174-4731-a696-219ae8b18a4d-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:28:18.456037 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.455888    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b760f7-9174-4731-a696-219ae8b18a4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71b760f7-9174-4731-a696-219ae8b18a4d" (UID: "71b760f7-9174-4731-a696-219ae8b18a4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:28:18.456307 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.456097    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b760f7-9174-4731-a696-219ae8b18a4d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71b760f7-9174-4731-a696-219ae8b18a4d" (UID: "71b760f7-9174-4731-a696-219ae8b18a4d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:28:18.457629 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.457603    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b760f7-9174-4731-a696-219ae8b18a4d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71b760f7-9174-4731-a696-219ae8b18a4d" (UID: "71b760f7-9174-4731-a696-219ae8b18a4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:28:18.457870 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.457849    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b760f7-9174-4731-a696-219ae8b18a4d-kube-api-access-5xlrx" (OuterVolumeSpecName: "kube-api-access-5xlrx") pod "71b760f7-9174-4731-a696-219ae8b18a4d" (UID: "71b760f7-9174-4731-a696-219ae8b18a4d"). InnerVolumeSpecName "kube-api-access-5xlrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:28:18.556576 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.556515    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xlrx\" (UniqueName: \"kubernetes.io/projected/71b760f7-9174-4731-a696-219ae8b18a4d-kube-api-access-5xlrx\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:28:18.556576 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.556540    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b760f7-9174-4731-a696-219ae8b18a4d-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:28:18.556576 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.556551    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b760f7-9174-4731-a696-219ae8b18a4d-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:28:18.556576 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:18.556560    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71b760f7-9174-4731-a696-219ae8b18a4d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:28:19.037574 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:19.037548    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:28:19.038047 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:19.038013    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 24 21:28:19.060060 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:19.060025    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t" event={"ID":"71b760f7-9174-4731-a696-219ae8b18a4d","Type":"ContainerDied","Data":"403123d412972125c665a221e9e9372b2ff646e7c9577c4e24887db5a1c524a3"}
Jul 24 21:28:19.060247 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:19.060073    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"
Jul 24 21:28:19.060247 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:19.060110    2560 scope.go:117] "RemoveContainer" containerID="a66b5d224748cbac0b011bb5f9169cb1fdc7b157c2503d62d130d2f91ad729df"
Jul 24 21:28:19.069514 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:19.069498    2560 scope.go:117] "RemoveContainer" containerID="fc9793ff90e6c532f469b922391db37b532bcb744606882f1f67ca231b046308"
Jul 24 21:28:19.078203 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:19.078186    2560 scope.go:117] "RemoveContainer" containerID="33c14045254bd95762b8518d1ca1aa5c8dc7f6f3bc7caa36b60beffc186f5bd8"
Jul 24 21:28:19.085935 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:19.085913    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"]
Jul 24 21:28:19.090048 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:19.090025    2560 scope.go:117] "RemoveContainer" containerID="01cb436745607abc5eb1f16ab83dbb88ced4e90c1dc93c6f237b631a8b0278b6"
Jul 24 21:28:19.090987 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:19.090919    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-05405-predictor-84cfbfbf4c-lgq8t"]
Jul 24 21:28:19.753866 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:19.753832    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" path="/var/lib/kubelet/pods/71b760f7-9174-4731-a696-219ae8b18a4d/volumes"
Jul 24 21:28:20.950388 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:20.950340    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 24 21:28:29.038742 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:29.038702    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 24 21:28:30.951012 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:30.950974    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 24 21:28:39.038729 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:39.038688    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 24 21:28:40.950613 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:40.950569    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 24 21:28:49.038560 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:49.038523    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 24 21:28:50.950397 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:50.950356    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 24 21:28:59.038267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:28:59.038229    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 24 21:29:00.951235 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:00.951198    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 24 21:29:02.749697 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:02.749659    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 24 21:29:09.039066 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:09.039039    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:29:12.750349 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:12.750318    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:29:27.767934 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:27.767901    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:29:28.164366 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:28.164322    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:29:37.802473 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:37.802438    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:29:37.988867 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:37.988831    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"]
Jul 24 21:29:37.989106 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:37.989084    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kserve-container" containerID="cri-o://c77ea050e6a58d4aef5503dcceb5e07bfd220c88d6c47747e5af8d8955a7039f" gracePeriod=30
Jul 24 21:29:37.989220 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:37.989149    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kube-rbac-proxy" containerID="cri-o://e4b22766f5210e702652754405ea210ba0d6d58b2d4753e8a2bf76bd46c2ce0b" gracePeriod=30
Jul 24 21:29:37.998022 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:37.997997    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:29:38.052114 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.052070    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"]
Jul 24 21:29:38.052645 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.052591    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent"
Jul 24 21:29:38.052645 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.052616    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent"
Jul 24 21:29:38.052645 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.052634    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="storage-initializer"
Jul 24 21:29:38.052645 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.052644    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="storage-initializer"
Jul 24 21:29:38.052858 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.052658    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container"
Jul 24 21:29:38.052858 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.052668    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container"
Jul 24 21:29:38.052858 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.052678    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kube-rbac-proxy"
Jul 24 21:29:38.052858 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.052686    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kube-rbac-proxy"
Jul 24 21:29:38.052858 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.052785    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="agent"
Jul 24 21:29:38.052858 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.052795    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kserve-container"
Jul 24 21:29:38.052858 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.052806    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b760f7-9174-4731-a696-219ae8b18a4d" containerName="kube-rbac-proxy"
Jul 24 21:29:38.055451 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.055428    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.059258 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.059236    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-6bdca-predictor-serving-cert\""
Jul 24 21:29:38.059389 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.059282    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-6bdca-kube-rbac-proxy-sar-config\""
Jul 24 21:29:38.062400 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.062370    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:29:38.066788 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.066766    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"]
Jul 24 21:29:38.133821 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.133791    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"]
Jul 24 21:29:38.135771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.135742    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:29:38.136330 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.136315    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.139203 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.139185    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-6bdca-predictor-serving-cert\""
Jul 24 21:29:38.139313 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.139225    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-6bdca-kube-rbac-proxy-sar-config\""
Jul 24 21:29:38.149849 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.149830    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"]
Jul 24 21:29:38.198069 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.198038    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"]
Jul 24 21:29:38.198374 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.198348    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kserve-container" containerID="cri-o://47e9ded3a5efc8a381b467213ed40f14d024dfc624bb6fb5f04bf1da54915df3" gracePeriod=30
Jul 24 21:29:38.198453 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.198373    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kube-rbac-proxy" containerID="cri-o://9f432ac84c600f47e6929833b31de314d3de6a5631f8b1203df0c5eb79c729b3" gracePeriod=30
Jul 24 21:29:38.210427 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.210401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dabc18d6-628b-4c48-9117-8bdb56c6bebd-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.210515 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.210457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.210582 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.210569    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dls2j\" (UniqueName: \"kubernetes.io/projected/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kube-api-access-dls2j\") pod \"isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.210637 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.210591    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.210637 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.210622    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dabc18d6-628b-4c48-9117-8bdb56c6bebd-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.210869 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.210643    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h72v\" (UniqueName: \"kubernetes.io/projected/23b050fc-7979-4bc1-998d-fda8656aab29-kube-api-access-6h72v\") pod \"isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.210869 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.210673    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b050fc-7979-4bc1-998d-fda8656aab29-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.210869 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.210743    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23b050fc-7979-4bc1-998d-fda8656aab29-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.210869 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.210786    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23b050fc-7979-4bc1-998d-fda8656aab29-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.210869 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.210824    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b050fc-7979-4bc1-998d-fda8656aab29-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.228033 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.228003    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:29:38.311392 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.311324    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.311392 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.311375    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dls2j\" (UniqueName: \"kubernetes.io/projected/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kube-api-access-dls2j\") pod \"isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.311392 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.311393    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.311670 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.311427    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dabc18d6-628b-4c48-9117-8bdb56c6bebd-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.311670 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.311592    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h72v\" (UniqueName: \"kubernetes.io/projected/23b050fc-7979-4bc1-998d-fda8656aab29-kube-api-access-6h72v\") pod \"isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.311670 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.311629    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b050fc-7979-4bc1-998d-fda8656aab29-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.311834 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.311675    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23b050fc-7979-4bc1-998d-fda8656aab29-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.311834 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.311720    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23b050fc-7979-4bc1-998d-fda8656aab29-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.312474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.311855    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.312474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.312006    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b050fc-7979-4bc1-998d-fda8656aab29-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.312474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.312085    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dabc18d6-628b-4c48-9117-8bdb56c6bebd-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.312474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.312168    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.312474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.312172    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dabc18d6-628b-4c48-9117-8bdb56c6bebd-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.312751 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.312487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23b050fc-7979-4bc1-998d-fda8656aab29-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.312751 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.312582    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b050fc-7979-4bc1-998d-fda8656aab29-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.312864 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.312842    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23b050fc-7979-4bc1-998d-fda8656aab29-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.314618 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.314594    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dabc18d6-628b-4c48-9117-8bdb56c6bebd-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.314864 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.314847    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b050fc-7979-4bc1-998d-fda8656aab29-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.321310 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.321289    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h72v\" (UniqueName: \"kubernetes.io/projected/23b050fc-7979-4bc1-998d-fda8656aab29-kube-api-access-6h72v\") pod \"isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.322816 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.322795    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dls2j\" (UniqueName: \"kubernetes.io/projected/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kube-api-access-dls2j\") pod \"isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.325575 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.325551    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:29:38.367195 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.367171    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:38.386822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.386799    2560 generic.go:356] "Generic (PLEG): container finished" podID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerID="e4b22766f5210e702652754405ea210ba0d6d58b2d4753e8a2bf76bd46c2ce0b" exitCode=2
Jul 24 21:29:38.386933 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.386872    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" event={"ID":"c0469501-f68e-4b73-99cf-4c0e36fbb284","Type":"ContainerDied","Data":"e4b22766f5210e702652754405ea210ba0d6d58b2d4753e8a2bf76bd46c2ce0b"}
Jul 24 21:29:38.389286 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.389266    2560 generic.go:356] "Generic (PLEG): container finished" podID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerID="9f432ac84c600f47e6929833b31de314d3de6a5631f8b1203df0c5eb79c729b3" exitCode=2
Jul 24 21:29:38.389364 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.389295    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" event={"ID":"4c6059f7-c136-4628-bc34-6403bbfe0eaf","Type":"ContainerDied","Data":"9f432ac84c600f47e6929833b31de314d3de6a5631f8b1203df0c5eb79c729b3"}
Jul 24 21:29:38.447302 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.447237    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:38.503442 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.503416    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"]
Jul 24 21:29:38.507066 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:29:38.507038    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b050fc_7979_4bc1_998d_fda8656aab29.slice/crio-a2a5c70b990625ca2d023f042a1e68376070f27306c6ec464c0507e3dcd3474d WatchSource:0}: Error finding container a2a5c70b990625ca2d023f042a1e68376070f27306c6ec464c0507e3dcd3474d: Status 404 returned error can't find the container with id a2a5c70b990625ca2d023f042a1e68376070f27306c6ec464c0507e3dcd3474d
Jul 24 21:29:38.590442 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:38.590367    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"]
Jul 24 21:29:38.593737 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:29:38.593710    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabc18d6_628b_4c48_9117_8bdb56c6bebd.slice/crio-d957cf49fe7374209023c13a1be463508329696f210856b24a04d5291ff65c38 WatchSource:0}: Error finding container d957cf49fe7374209023c13a1be463508329696f210856b24a04d5291ff65c38: Status 404 returned error can't find the container with id d957cf49fe7374209023c13a1be463508329696f210856b24a04d5291ff65c38
Jul 24 21:29:39.033548 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:39.033511    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 24 21:29:39.039113 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:39.039080    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 24 21:29:39.395182 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:39.395153    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" event={"ID":"23b050fc-7979-4bc1-998d-fda8656aab29","Type":"ContainerStarted","Data":"c3362cc0860f403b02149f89141af1a70e7f4bfa0a2ea2790bb5539de9373746"}
Jul 24 21:29:39.395385 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:39.395219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" event={"ID":"23b050fc-7979-4bc1-998d-fda8656aab29","Type":"ContainerStarted","Data":"a2a5c70b990625ca2d023f042a1e68376070f27306c6ec464c0507e3dcd3474d"}
Jul 24 21:29:39.396676 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:39.396650    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" event={"ID":"dabc18d6-628b-4c48-9117-8bdb56c6bebd","Type":"ContainerStarted","Data":"ba04f9641b0cdcef80657ae61b4586a4dca5bcc4dc006f98f4fda6dac6b5ea15"}
Jul 24 21:29:39.396796 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:39.396703    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" event={"ID":"dabc18d6-628b-4c48-9117-8bdb56c6bebd","Type":"ContainerStarted","Data":"d957cf49fe7374209023c13a1be463508329696f210856b24a04d5291ff65c38"}
Jul 24 21:29:40.945573 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:40.945528    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 24 21:29:41.650276 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.650251    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:29:41.745032 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.744940    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6059f7-c136-4628-bc34-6403bbfe0eaf-proxy-tls\") pod \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") "
Jul 24 21:29:41.745032 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.744982    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kube-rbac-proxy-sar-config\") pod \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") "
Jul 24 21:29:41.745032 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.744999    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7d89\" (UniqueName: \"kubernetes.io/projected/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kube-api-access-g7d89\") pod \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") "
Jul 24 21:29:41.745032 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.745027    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kserve-provision-location\") pod \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") "
Jul 24 21:29:41.745421 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.745067    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c6059f7-c136-4628-bc34-6403bbfe0eaf-cabundle-cert\") pod \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\" (UID: \"4c6059f7-c136-4628-bc34-6403bbfe0eaf\") "
Jul 24 21:29:41.745421 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.745396    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4c6059f7-c136-4628-bc34-6403bbfe0eaf" (UID: "4c6059f7-c136-4628-bc34-6403bbfe0eaf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:29:41.745502 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.745448    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4c6059f7-c136-4628-bc34-6403bbfe0eaf" (UID: "4c6059f7-c136-4628-bc34-6403bbfe0eaf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:29:41.745560 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.745534    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6059f7-c136-4628-bc34-6403bbfe0eaf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4c6059f7-c136-4628-bc34-6403bbfe0eaf" (UID: "4c6059f7-c136-4628-bc34-6403bbfe0eaf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:29:41.747279 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.747256    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c6059f7-c136-4628-bc34-6403bbfe0eaf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4c6059f7-c136-4628-bc34-6403bbfe0eaf" (UID: "4c6059f7-c136-4628-bc34-6403bbfe0eaf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:29:41.747279 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.747264    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kube-api-access-g7d89" (OuterVolumeSpecName: "kube-api-access-g7d89") pod "4c6059f7-c136-4628-bc34-6403bbfe0eaf" (UID: "4c6059f7-c136-4628-bc34-6403bbfe0eaf"). InnerVolumeSpecName "kube-api-access-g7d89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:29:41.845950 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.845917    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c6059f7-c136-4628-bc34-6403bbfe0eaf-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:29:41.845950 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.845941    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:29:41.845950 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.845950    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7d89\" (UniqueName: \"kubernetes.io/projected/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kube-api-access-g7d89\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:29:41.845950 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.845960    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c6059f7-c136-4628-bc34-6403bbfe0eaf-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:29:41.846346 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:41.845969    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c6059f7-c136-4628-bc34-6403bbfe0eaf-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:29:42.411015 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.410978    2560 generic.go:356] "Generic (PLEG): container finished" podID="23b050fc-7979-4bc1-998d-fda8656aab29" containerID="c3362cc0860f403b02149f89141af1a70e7f4bfa0a2ea2790bb5539de9373746" exitCode=0
Jul 24 21:29:42.411438 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.411051    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" event={"ID":"23b050fc-7979-4bc1-998d-fda8656aab29","Type":"ContainerDied","Data":"c3362cc0860f403b02149f89141af1a70e7f4bfa0a2ea2790bb5539de9373746"}
Jul 24 21:29:42.413192 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.413164    2560 generic.go:356] "Generic (PLEG): container finished" podID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerID="47e9ded3a5efc8a381b467213ed40f14d024dfc624bb6fb5f04bf1da54915df3" exitCode=0
Jul 24 21:29:42.413322 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.413225    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"
Jul 24 21:29:42.413322 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.413249    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" event={"ID":"4c6059f7-c136-4628-bc34-6403bbfe0eaf","Type":"ContainerDied","Data":"47e9ded3a5efc8a381b467213ed40f14d024dfc624bb6fb5f04bf1da54915df3"}
Jul 24 21:29:42.413322 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.413304    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm" event={"ID":"4c6059f7-c136-4628-bc34-6403bbfe0eaf","Type":"ContainerDied","Data":"dda55dee947584e00d52beb7a1ff0e94997c46f60befcd8bd9f01ef676bf0cbd"}
Jul 24 21:29:42.413555 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.413331    2560 scope.go:117] "RemoveContainer" containerID="9f432ac84c600f47e6929833b31de314d3de6a5631f8b1203df0c5eb79c729b3"
Jul 24 21:29:42.422166 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.422150    2560 scope.go:117] "RemoveContainer" containerID="47e9ded3a5efc8a381b467213ed40f14d024dfc624bb6fb5f04bf1da54915df3"
Jul 24 21:29:42.430016 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.429999    2560 scope.go:117] "RemoveContainer" containerID="434a11f4d70d5a2acef884a3d1b518d506a7fd8df8b36219e5467953fb1c4e1c"
Jul 24 21:29:42.444839 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.444817    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"]
Jul 24 21:29:42.447889 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.447868    2560 scope.go:117] "RemoveContainer" containerID="9f432ac84c600f47e6929833b31de314d3de6a5631f8b1203df0c5eb79c729b3"
Jul 24 21:29:42.448237 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:29:42.448215    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f432ac84c600f47e6929833b31de314d3de6a5631f8b1203df0c5eb79c729b3\": container with ID starting with 9f432ac84c600f47e6929833b31de314d3de6a5631f8b1203df0c5eb79c729b3 not found: ID does not exist" containerID="9f432ac84c600f47e6929833b31de314d3de6a5631f8b1203df0c5eb79c729b3"
Jul 24 21:29:42.448318 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.448244    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-8100c-predictor-6577d79b6f-z5rmm"]
Jul 24 21:29:42.448318 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.448243    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f432ac84c600f47e6929833b31de314d3de6a5631f8b1203df0c5eb79c729b3"} err="failed to get container status \"9f432ac84c600f47e6929833b31de314d3de6a5631f8b1203df0c5eb79c729b3\": rpc error: code = NotFound desc = could not find container \"9f432ac84c600f47e6929833b31de314d3de6a5631f8b1203df0c5eb79c729b3\": container with ID starting with 9f432ac84c600f47e6929833b31de314d3de6a5631f8b1203df0c5eb79c729b3 not found: ID does not exist"
Jul 24 21:29:42.448318 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.448296    2560 scope.go:117] "RemoveContainer" containerID="47e9ded3a5efc8a381b467213ed40f14d024dfc624bb6fb5f04bf1da54915df3"
Jul 24 21:29:42.448667 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:29:42.448640    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e9ded3a5efc8a381b467213ed40f14d024dfc624bb6fb5f04bf1da54915df3\": container with ID starting with 47e9ded3a5efc8a381b467213ed40f14d024dfc624bb6fb5f04bf1da54915df3 not found: ID does not exist" containerID="47e9ded3a5efc8a381b467213ed40f14d024dfc624bb6fb5f04bf1da54915df3"
Jul 24 21:29:42.448802 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.448773    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e9ded3a5efc8a381b467213ed40f14d024dfc624bb6fb5f04bf1da54915df3"} err="failed to get container status \"47e9ded3a5efc8a381b467213ed40f14d024dfc624bb6fb5f04bf1da54915df3\": rpc error: code = NotFound desc = could not find container \"47e9ded3a5efc8a381b467213ed40f14d024dfc624bb6fb5f04bf1da54915df3\": container with ID starting with 47e9ded3a5efc8a381b467213ed40f14d024dfc624bb6fb5f04bf1da54915df3 not found: ID does not exist"
Jul 24 21:29:42.448921 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.448805    2560 scope.go:117] "RemoveContainer" containerID="434a11f4d70d5a2acef884a3d1b518d506a7fd8df8b36219e5467953fb1c4e1c"
Jul 24 21:29:42.449157 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:29:42.449116    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"434a11f4d70d5a2acef884a3d1b518d506a7fd8df8b36219e5467953fb1c4e1c\": container with ID starting with 434a11f4d70d5a2acef884a3d1b518d506a7fd8df8b36219e5467953fb1c4e1c not found: ID does not exist" containerID="434a11f4d70d5a2acef884a3d1b518d506a7fd8df8b36219e5467953fb1c4e1c"
Jul 24 21:29:42.449227 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.449169    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"434a11f4d70d5a2acef884a3d1b518d506a7fd8df8b36219e5467953fb1c4e1c"} err="failed to get container status \"434a11f4d70d5a2acef884a3d1b518d506a7fd8df8b36219e5467953fb1c4e1c\": rpc error: code = NotFound desc = could not find container \"434a11f4d70d5a2acef884a3d1b518d506a7fd8df8b36219e5467953fb1c4e1c\": container with ID starting with 434a11f4d70d5a2acef884a3d1b518d506a7fd8df8b36219e5467953fb1c4e1c not found: ID does not exist"
Jul 24 21:29:42.628953 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.628934    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:29:42.752342 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.752253    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0469501-f68e-4b73-99cf-4c0e36fbb284-kube-rbac-proxy-sar-config\") pod \"c0469501-f68e-4b73-99cf-4c0e36fbb284\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") "
Jul 24 21:29:42.752342 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.752287    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0469501-f68e-4b73-99cf-4c0e36fbb284-cabundle-cert\") pod \"c0469501-f68e-4b73-99cf-4c0e36fbb284\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") "
Jul 24 21:29:42.752342 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.752325    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0469501-f68e-4b73-99cf-4c0e36fbb284-proxy-tls\") pod \"c0469501-f68e-4b73-99cf-4c0e36fbb284\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") "
Jul 24 21:29:42.752639 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.752366    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzb5x\" (UniqueName: \"kubernetes.io/projected/c0469501-f68e-4b73-99cf-4c0e36fbb284-kube-api-access-dzb5x\") pod \"c0469501-f68e-4b73-99cf-4c0e36fbb284\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") "
Jul 24 21:29:42.752639 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.752410    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0469501-f68e-4b73-99cf-4c0e36fbb284-kserve-provision-location\") pod \"c0469501-f68e-4b73-99cf-4c0e36fbb284\" (UID: \"c0469501-f68e-4b73-99cf-4c0e36fbb284\") "
Jul 24 21:29:42.752734 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.752659    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0469501-f68e-4b73-99cf-4c0e36fbb284-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c0469501-f68e-4b73-99cf-4c0e36fbb284" (UID: "c0469501-f68e-4b73-99cf-4c0e36fbb284"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:29:42.752734 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.752714    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0469501-f68e-4b73-99cf-4c0e36fbb284-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c0469501-f68e-4b73-99cf-4c0e36fbb284" (UID: "c0469501-f68e-4b73-99cf-4c0e36fbb284"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:29:42.752804 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.752736    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0469501-f68e-4b73-99cf-4c0e36fbb284-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c0469501-f68e-4b73-99cf-4c0e36fbb284" (UID: "c0469501-f68e-4b73-99cf-4c0e36fbb284"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:29:42.754546 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.754518    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0469501-f68e-4b73-99cf-4c0e36fbb284-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c0469501-f68e-4b73-99cf-4c0e36fbb284" (UID: "c0469501-f68e-4b73-99cf-4c0e36fbb284"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:29:42.754546 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.754529    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0469501-f68e-4b73-99cf-4c0e36fbb284-kube-api-access-dzb5x" (OuterVolumeSpecName: "kube-api-access-dzb5x") pod "c0469501-f68e-4b73-99cf-4c0e36fbb284" (UID: "c0469501-f68e-4b73-99cf-4c0e36fbb284"). InnerVolumeSpecName "kube-api-access-dzb5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:29:42.853668 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.853630    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0469501-f68e-4b73-99cf-4c0e36fbb284-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:29:42.853668 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.853660    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0469501-f68e-4b73-99cf-4c0e36fbb284-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:29:42.853840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.853675    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0469501-f68e-4b73-99cf-4c0e36fbb284-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:29:42.853840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.853690    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0469501-f68e-4b73-99cf-4c0e36fbb284-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:29:42.853840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:42.853703    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzb5x\" (UniqueName: \"kubernetes.io/projected/c0469501-f68e-4b73-99cf-4c0e36fbb284-kube-api-access-dzb5x\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:29:43.425664 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.425627    2560 generic.go:356] "Generic (PLEG): container finished" podID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerID="ba04f9641b0cdcef80657ae61b4586a4dca5bcc4dc006f98f4fda6dac6b5ea15" exitCode=0
Jul 24 21:29:43.426082 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.425701    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" event={"ID":"dabc18d6-628b-4c48-9117-8bdb56c6bebd","Type":"ContainerDied","Data":"ba04f9641b0cdcef80657ae61b4586a4dca5bcc4dc006f98f4fda6dac6b5ea15"}
Jul 24 21:29:43.427681 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.427654    2560 generic.go:356] "Generic (PLEG): container finished" podID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerID="c77ea050e6a58d4aef5503dcceb5e07bfd220c88d6c47747e5af8d8955a7039f" exitCode=0
Jul 24 21:29:43.427777 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.427718    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"
Jul 24 21:29:43.427777 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.427731    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" event={"ID":"c0469501-f68e-4b73-99cf-4c0e36fbb284","Type":"ContainerDied","Data":"c77ea050e6a58d4aef5503dcceb5e07bfd220c88d6c47747e5af8d8955a7039f"}
Jul 24 21:29:43.427777 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.427776    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd" event={"ID":"c0469501-f68e-4b73-99cf-4c0e36fbb284","Type":"ContainerDied","Data":"3767f956f23796cd2b4275a6b873858e039f27725ebbfb4d424016fa0a5ef475"}
Jul 24 21:29:43.427911 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.427810    2560 scope.go:117] "RemoveContainer" containerID="e4b22766f5210e702652754405ea210ba0d6d58b2d4753e8a2bf76bd46c2ce0b"
Jul 24 21:29:43.430225 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.430193    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" event={"ID":"23b050fc-7979-4bc1-998d-fda8656aab29","Type":"ContainerStarted","Data":"6a9d9fb60536f9c549f7cba4925ec3b6fe6a11bce658514c10d6ad49036d78b5"}
Jul 24 21:29:43.430378 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.430251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" event={"ID":"23b050fc-7979-4bc1-998d-fda8656aab29","Type":"ContainerStarted","Data":"4ae35e7a8a6923ac9e75990e21fdbf22ba34f80208fe2ad233463f2ad9c63ea8"}
Jul 24 21:29:43.430526 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.430501    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:43.439670 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.439654    2560 scope.go:117] "RemoveContainer" containerID="c77ea050e6a58d4aef5503dcceb5e07bfd220c88d6c47747e5af8d8955a7039f"
Jul 24 21:29:43.447931 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.447916    2560 scope.go:117] "RemoveContainer" containerID="8b4a51da16ff691adb2e46c2fbc42aacca0868b589de970681286e3256b3b384"
Jul 24 21:29:43.459690 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.459648    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"]
Jul 24 21:29:43.464717 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.464673    2560 scope.go:117] "RemoveContainer" containerID="e4b22766f5210e702652754405ea210ba0d6d58b2d4753e8a2bf76bd46c2ce0b"
Jul 24 21:29:43.464987 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:29:43.464968    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b22766f5210e702652754405ea210ba0d6d58b2d4753e8a2bf76bd46c2ce0b\": container with ID starting with e4b22766f5210e702652754405ea210ba0d6d58b2d4753e8a2bf76bd46c2ce0b not found: ID does not exist" containerID="e4b22766f5210e702652754405ea210ba0d6d58b2d4753e8a2bf76bd46c2ce0b"
Jul 24 21:29:43.465063 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.464994    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b22766f5210e702652754405ea210ba0d6d58b2d4753e8a2bf76bd46c2ce0b"} err="failed to get container status \"e4b22766f5210e702652754405ea210ba0d6d58b2d4753e8a2bf76bd46c2ce0b\": rpc error: code = NotFound desc = could not find container \"e4b22766f5210e702652754405ea210ba0d6d58b2d4753e8a2bf76bd46c2ce0b\": container with ID starting with e4b22766f5210e702652754405ea210ba0d6d58b2d4753e8a2bf76bd46c2ce0b not found: ID does not exist"
Jul 24 21:29:43.465063 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.465011    2560 scope.go:117] "RemoveContainer" containerID="c77ea050e6a58d4aef5503dcceb5e07bfd220c88d6c47747e5af8d8955a7039f"
Jul 24 21:29:43.465295 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:29:43.465273    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c77ea050e6a58d4aef5503dcceb5e07bfd220c88d6c47747e5af8d8955a7039f\": container with ID starting with c77ea050e6a58d4aef5503dcceb5e07bfd220c88d6c47747e5af8d8955a7039f not found: ID does not exist" containerID="c77ea050e6a58d4aef5503dcceb5e07bfd220c88d6c47747e5af8d8955a7039f"
Jul 24 21:29:43.465362 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.465313    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-8100c-predictor-76445879cf-7g4nd"]
Jul 24 21:29:43.465362 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.465310    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c77ea050e6a58d4aef5503dcceb5e07bfd220c88d6c47747e5af8d8955a7039f"} err="failed to get container status \"c77ea050e6a58d4aef5503dcceb5e07bfd220c88d6c47747e5af8d8955a7039f\": rpc error: code = NotFound desc = could not find container \"c77ea050e6a58d4aef5503dcceb5e07bfd220c88d6c47747e5af8d8955a7039f\": container with ID starting with c77ea050e6a58d4aef5503dcceb5e07bfd220c88d6c47747e5af8d8955a7039f not found: ID does not exist"
Jul 24 21:29:43.465464 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.465368    2560 scope.go:117] "RemoveContainer" containerID="8b4a51da16ff691adb2e46c2fbc42aacca0868b589de970681286e3256b3b384"
Jul 24 21:29:43.465735 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:29:43.465706    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b4a51da16ff691adb2e46c2fbc42aacca0868b589de970681286e3256b3b384\": container with ID starting with 8b4a51da16ff691adb2e46c2fbc42aacca0868b589de970681286e3256b3b384 not found: ID does not exist" containerID="8b4a51da16ff691adb2e46c2fbc42aacca0868b589de970681286e3256b3b384"
Jul 24 21:29:43.465831 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.465743    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4a51da16ff691adb2e46c2fbc42aacca0868b589de970681286e3256b3b384"} err="failed to get container status \"8b4a51da16ff691adb2e46c2fbc42aacca0868b589de970681286e3256b3b384\": rpc error: code = NotFound desc = could not find container \"8b4a51da16ff691adb2e46c2fbc42aacca0868b589de970681286e3256b3b384\": container with ID starting with 8b4a51da16ff691adb2e46c2fbc42aacca0868b589de970681286e3256b3b384 not found: ID does not exist"
Jul 24 21:29:43.482633 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.482591    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" podStartSLOduration=1.939328342 podStartE2EDuration="5.482578273s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.543249931s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:29:43.481029205 +0000 UTC m=+862.237375018" watchObservedRunningTime="2026-07-24 21:29:43.482578273 +0000 UTC m=+862.238924086"
Jul 24 21:29:43.753883 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.753805    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" path="/var/lib/kubelet/pods/4c6059f7-c136-4628-bc34-6403bbfe0eaf/volumes"
Jul 24 21:29:43.754349 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:43.754334    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" path="/var/lib/kubelet/pods/c0469501-f68e-4b73-99cf-4c0e36fbb284/volumes"
Jul 24 21:29:44.437061 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:44.437019    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" event={"ID":"dabc18d6-628b-4c48-9117-8bdb56c6bebd","Type":"ContainerStarted","Data":"20fc7a6c042bca74f50bc3ef11fba42637cd66123f64d60cdda3164b8dd997a5"}
Jul 24 21:29:44.437530 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:44.437096    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" event={"ID":"dabc18d6-628b-4c48-9117-8bdb56c6bebd","Type":"ContainerStarted","Data":"3d081088b657709ce193f096e1242fba133ba7bb0cecc9c7bab736a7bd3f2141"}
Jul 24 21:29:44.437530 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:44.437333    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:44.438241 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:44.438221    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:44.439479 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:44.439456    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 24 21:29:44.459895 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:44.459859    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" podStartSLOduration=2.5485391120000003 podStartE2EDuration="6.459845718s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.911306606s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:29:44.458277169 +0000 UTC m=+863.214622982" watchObservedRunningTime="2026-07-24 21:29:44.459845718 +0000 UTC m=+863.216191532"
Jul 24 21:29:45.442586 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:45.442546    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 24 21:29:45.442586 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:45.442569    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:45.443724 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:45.443703    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:29:46.446663 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:46.446628    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:29:50.447158 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:50.447111    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:29:50.447567 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:50.447543    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 24 21:29:51.451285 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:51.451257    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:29:51.451859 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:29:51.451834    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:30:00.447681 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:30:00.447637    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 24 21:30:01.452541 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:30:01.452499    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:30:10.448384 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:30:10.448341    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 24 21:30:11.452611 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:30:11.452574    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:30:20.448364 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:30:20.448327    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 24 21:30:21.451855 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:30:21.451819    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:30:21.667490 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:30:21.667466    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:30:21.674911 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:30:21.674891    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:30:24.512639 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:30:24.512602    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:30:24.567121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:30:24.567091    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:30:30.448169 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:30:30.448107    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 24 21:30:31.452668 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:30:31.452632    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:30:40.448358 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:30:40.448314    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 24 21:30:41.452714 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:30:41.452687    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:30:50.448372 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:30:50.448330    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 24 21:31:00.448325 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:00.448292    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:31:18.081904 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:18.081872    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:31:19.372089 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:19.372053    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:31:28.151278 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.151247    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:31:28.312390 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.312361    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"]
Jul 24 21:31:28.312628 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.312577    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kserve-container" containerID="cri-o://4ae35e7a8a6923ac9e75990e21fdbf22ba34f80208fe2ad233463f2ad9c63ea8" gracePeriod=30
Jul 24 21:31:28.312771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.312633    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kube-rbac-proxy" containerID="cri-o://6a9d9fb60536f9c549f7cba4925ec3b6fe6a11bce658514c10d6ad49036d78b5" gracePeriod=30
Jul 24 21:31:28.315706 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.315682    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:31:28.369966 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.369938    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"]
Jul 24 21:31:28.370312 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370299    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kube-rbac-proxy"
Jul 24 21:31:28.370360 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370314    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kube-rbac-proxy"
Jul 24 21:31:28.370360 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370334    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="storage-initializer"
Jul 24 21:31:28.370360 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370339    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="storage-initializer"
Jul 24 21:31:28.370360 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370348    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="storage-initializer"
Jul 24 21:31:28.370360 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370353    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="storage-initializer"
Jul 24 21:31:28.370360 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370359    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kserve-container"
Jul 24 21:31:28.370539 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370364    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kserve-container"
Jul 24 21:31:28.370539 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370370    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kube-rbac-proxy"
Jul 24 21:31:28.370539 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370375    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kube-rbac-proxy"
Jul 24 21:31:28.370539 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370381    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kserve-container"
Jul 24 21:31:28.370539 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370386    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kserve-container"
Jul 24 21:31:28.370539 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370436    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kserve-container"
Jul 24 21:31:28.370539 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370443    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kube-rbac-proxy"
Jul 24 21:31:28.370539 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370451    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c6059f7-c136-4628-bc34-6403bbfe0eaf" containerName="kube-rbac-proxy"
Jul 24 21:31:28.370539 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.370458    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0469501-f68e-4b73-99cf-4c0e36fbb284" containerName="kserve-container"
Jul 24 21:31:28.373530 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.373512    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:31:28.375733 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.375716    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-103c4-predictor-serving-cert\""
Jul 24 21:31:28.375855 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.375835    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-103c4-kube-rbac-proxy-sar-config\""
Jul 24 21:31:28.384439 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.384418    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"]
Jul 24 21:31:28.389994 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.389977    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:31:28.472580 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.472551    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"]
Jul 24 21:31:28.472825 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.472799    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kserve-container" containerID="cri-o://3d081088b657709ce193f096e1242fba133ba7bb0cecc9c7bab736a7bd3f2141" gracePeriod=30
Jul 24 21:31:28.472922 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.472852    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kube-rbac-proxy" containerID="cri-o://20fc7a6c042bca74f50bc3ef11fba42637cd66123f64d60cdda3164b8dd997a5" gracePeriod=30
Jul 24 21:31:28.481487 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.481455    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:31:28.558433 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.558405    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m54vg\" (UniqueName: \"kubernetes.io/projected/8a58d115-6917-4419-b105-6e5498a2e959-kube-api-access-m54vg\") pod \"message-dumper-raw-103c4-predictor-86555d876f-r7wbl\" (UID: \"8a58d115-6917-4419-b105-6e5498a2e959\") " pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:31:28.558549 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.558463    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a58d115-6917-4419-b105-6e5498a2e959-proxy-tls\") pod \"message-dumper-raw-103c4-predictor-86555d876f-r7wbl\" (UID: \"8a58d115-6917-4419-b105-6e5498a2e959\") " pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:31:28.558619 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.558559    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a58d115-6917-4419-b105-6e5498a2e959-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-103c4-predictor-86555d876f-r7wbl\" (UID: \"8a58d115-6917-4419-b105-6e5498a2e959\") " pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:31:28.615700 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.615671    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:31:28.659444 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.659417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a58d115-6917-4419-b105-6e5498a2e959-proxy-tls\") pod \"message-dumper-raw-103c4-predictor-86555d876f-r7wbl\" (UID: \"8a58d115-6917-4419-b105-6e5498a2e959\") " pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:31:28.659547 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.659472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a58d115-6917-4419-b105-6e5498a2e959-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-103c4-predictor-86555d876f-r7wbl\" (UID: \"8a58d115-6917-4419-b105-6e5498a2e959\") " pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:31:28.659547 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.659538    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m54vg\" (UniqueName: \"kubernetes.io/projected/8a58d115-6917-4419-b105-6e5498a2e959-kube-api-access-m54vg\") pod \"message-dumper-raw-103c4-predictor-86555d876f-r7wbl\" (UID: \"8a58d115-6917-4419-b105-6e5498a2e959\") " pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:31:28.660071 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.660052    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a58d115-6917-4419-b105-6e5498a2e959-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-103c4-predictor-86555d876f-r7wbl\" (UID: \"8a58d115-6917-4419-b105-6e5498a2e959\") " pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:31:28.661784 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.661760    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a58d115-6917-4419-b105-6e5498a2e959-proxy-tls\") pod \"message-dumper-raw-103c4-predictor-86555d876f-r7wbl\" (UID: \"8a58d115-6917-4419-b105-6e5498a2e959\") " pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:31:28.668117 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.668097    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m54vg\" (UniqueName: \"kubernetes.io/projected/8a58d115-6917-4419-b105-6e5498a2e959-kube-api-access-m54vg\") pod \"message-dumper-raw-103c4-predictor-86555d876f-r7wbl\" (UID: \"8a58d115-6917-4419-b105-6e5498a2e959\") " pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:31:28.683964 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.683912    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:31:28.812606 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.812541    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"]
Jul 24 21:31:28.815183 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:31:28.815158    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a58d115_6917_4419_b105_6e5498a2e959.slice/crio-486cb23a6467e3c2b547220a0f799dd9c5c5c473cca6dbf9d089d20567ef54e8 WatchSource:0}: Error finding container 486cb23a6467e3c2b547220a0f799dd9c5c5c473cca6dbf9d089d20567ef54e8: Status 404 returned error can't find the container with id 486cb23a6467e3c2b547220a0f799dd9c5c5c473cca6dbf9d089d20567ef54e8
Jul 24 21:31:28.859268 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.859232    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl" event={"ID":"8a58d115-6917-4419-b105-6e5498a2e959","Type":"ContainerStarted","Data":"486cb23a6467e3c2b547220a0f799dd9c5c5c473cca6dbf9d089d20567ef54e8"}
Jul 24 21:31:28.861441 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.861418    2560 generic.go:356] "Generic (PLEG): container finished" podID="23b050fc-7979-4bc1-998d-fda8656aab29" containerID="6a9d9fb60536f9c549f7cba4925ec3b6fe6a11bce658514c10d6ad49036d78b5" exitCode=2
Jul 24 21:31:28.861526 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.861493    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" event={"ID":"23b050fc-7979-4bc1-998d-fda8656aab29","Type":"ContainerDied","Data":"6a9d9fb60536f9c549f7cba4925ec3b6fe6a11bce658514c10d6ad49036d78b5"}
Jul 24 21:31:28.863527 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.863510    2560 generic.go:356] "Generic (PLEG): container finished" podID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerID="20fc7a6c042bca74f50bc3ef11fba42637cd66123f64d60cdda3164b8dd997a5" exitCode=2
Jul 24 21:31:28.863601 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:28.863545    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" event={"ID":"dabc18d6-628b-4c48-9117-8bdb56c6bebd","Type":"ContainerDied","Data":"20fc7a6c042bca74f50bc3ef11fba42637cd66123f64d60cdda3164b8dd997a5"}
Jul 24 21:31:30.443546 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:30.443502    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 24 21:31:30.447840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:30.447817    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 24 21:31:30.873870 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:30.873833    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl" event={"ID":"8a58d115-6917-4419-b105-6e5498a2e959","Type":"ContainerStarted","Data":"6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd"}
Jul 24 21:31:30.874083 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:30.873902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl" event={"ID":"8a58d115-6917-4419-b105-6e5498a2e959","Type":"ContainerStarted","Data":"28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3"}
Jul 24 21:31:30.874083 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:30.873940    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:31:30.874083 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:30.873971    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:31:30.875712 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:30.875690    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:31:30.891903 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:30.891861    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl" podStartSLOduration=1.646342221 podStartE2EDuration="2.891846628s" totalImagesPullingTime="1.245504407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:31:30.891431695 +0000 UTC m=+969.647777508" watchObservedRunningTime="2026-07-24 21:31:30.891846628 +0000 UTC m=+969.648192441"
Jul 24 21:31:31.446893 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:31.446855    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 24 21:31:31.452045 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:31.452017    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 24 21:31:32.121237 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.121211    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:31:32.188905 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.188832    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kserve-provision-location\") pod \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") "
Jul 24 21:31:32.188905 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.188874    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dls2j\" (UniqueName: \"kubernetes.io/projected/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kube-api-access-dls2j\") pod \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") "
Jul 24 21:31:32.188905 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.188894    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kube-rbac-proxy-sar-config\") pod \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") "
Jul 24 21:31:32.189191 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.188923    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dabc18d6-628b-4c48-9117-8bdb56c6bebd-cabundle-cert\") pod \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") "
Jul 24 21:31:32.189191 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.188942    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dabc18d6-628b-4c48-9117-8bdb56c6bebd-proxy-tls\") pod \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\" (UID: \"dabc18d6-628b-4c48-9117-8bdb56c6bebd\") "
Jul 24 21:31:32.189315 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.189187    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dabc18d6-628b-4c48-9117-8bdb56c6bebd" (UID: "dabc18d6-628b-4c48-9117-8bdb56c6bebd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:31:32.189315 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.189292    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dabc18d6-628b-4c48-9117-8bdb56c6bebd" (UID: "dabc18d6-628b-4c48-9117-8bdb56c6bebd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:31:32.189398 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.189322    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dabc18d6-628b-4c48-9117-8bdb56c6bebd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dabc18d6-628b-4c48-9117-8bdb56c6bebd" (UID: "dabc18d6-628b-4c48-9117-8bdb56c6bebd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:31:32.191091 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.191073    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kube-api-access-dls2j" (OuterVolumeSpecName: "kube-api-access-dls2j") pod "dabc18d6-628b-4c48-9117-8bdb56c6bebd" (UID: "dabc18d6-628b-4c48-9117-8bdb56c6bebd"). InnerVolumeSpecName "kube-api-access-dls2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:31:32.191184 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.191142    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dabc18d6-628b-4c48-9117-8bdb56c6bebd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dabc18d6-628b-4c48-9117-8bdb56c6bebd" (UID: "dabc18d6-628b-4c48-9117-8bdb56c6bebd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:31:32.290393 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.290359    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:31:32.290393 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.290387    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dls2j\" (UniqueName: \"kubernetes.io/projected/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kube-api-access-dls2j\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:31:32.290393 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.290396    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dabc18d6-628b-4c48-9117-8bdb56c6bebd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:31:32.290611 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.290406    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dabc18d6-628b-4c48-9117-8bdb56c6bebd-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:31:32.290611 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.290416    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dabc18d6-628b-4c48-9117-8bdb56c6bebd-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:31:32.886809 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.886782    2560 generic.go:356] "Generic (PLEG): container finished" podID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerID="3d081088b657709ce193f096e1242fba133ba7bb0cecc9c7bab736a7bd3f2141" exitCode=0
Jul 24 21:31:32.887277 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.886828    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" event={"ID":"dabc18d6-628b-4c48-9117-8bdb56c6bebd","Type":"ContainerDied","Data":"3d081088b657709ce193f096e1242fba133ba7bb0cecc9c7bab736a7bd3f2141"}
Jul 24 21:31:32.887277 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.886844    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"
Jul 24 21:31:32.887277 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.886902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24" event={"ID":"dabc18d6-628b-4c48-9117-8bdb56c6bebd","Type":"ContainerDied","Data":"d957cf49fe7374209023c13a1be463508329696f210856b24a04d5291ff65c38"}
Jul 24 21:31:32.887277 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.886942    2560 scope.go:117] "RemoveContainer" containerID="20fc7a6c042bca74f50bc3ef11fba42637cd66123f64d60cdda3164b8dd997a5"
Jul 24 21:31:32.889628 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.889601    2560 generic.go:356] "Generic (PLEG): container finished" podID="23b050fc-7979-4bc1-998d-fda8656aab29" containerID="4ae35e7a8a6923ac9e75990e21fdbf22ba34f80208fe2ad233463f2ad9c63ea8" exitCode=0
Jul 24 21:31:32.890162 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.890139    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" event={"ID":"23b050fc-7979-4bc1-998d-fda8656aab29","Type":"ContainerDied","Data":"4ae35e7a8a6923ac9e75990e21fdbf22ba34f80208fe2ad233463f2ad9c63ea8"}
Jul 24 21:31:32.896745 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.896718    2560 scope.go:117] "RemoveContainer" containerID="3d081088b657709ce193f096e1242fba133ba7bb0cecc9c7bab736a7bd3f2141"
Jul 24 21:31:32.904289 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.904272    2560 scope.go:117] "RemoveContainer" containerID="ba04f9641b0cdcef80657ae61b4586a4dca5bcc4dc006f98f4fda6dac6b5ea15"
Jul 24 21:31:32.911456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.911439    2560 scope.go:117] "RemoveContainer" containerID="20fc7a6c042bca74f50bc3ef11fba42637cd66123f64d60cdda3164b8dd997a5"
Jul 24 21:31:32.911723 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:31:32.911708    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20fc7a6c042bca74f50bc3ef11fba42637cd66123f64d60cdda3164b8dd997a5\": container with ID starting with 20fc7a6c042bca74f50bc3ef11fba42637cd66123f64d60cdda3164b8dd997a5 not found: ID does not exist" containerID="20fc7a6c042bca74f50bc3ef11fba42637cd66123f64d60cdda3164b8dd997a5"
Jul 24 21:31:32.911778 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.911732    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20fc7a6c042bca74f50bc3ef11fba42637cd66123f64d60cdda3164b8dd997a5"} err="failed to get container status \"20fc7a6c042bca74f50bc3ef11fba42637cd66123f64d60cdda3164b8dd997a5\": rpc error: code = NotFound desc = could not find container \"20fc7a6c042bca74f50bc3ef11fba42637cd66123f64d60cdda3164b8dd997a5\": container with ID starting with 20fc7a6c042bca74f50bc3ef11fba42637cd66123f64d60cdda3164b8dd997a5 not found: ID does not exist"
Jul 24 21:31:32.911778 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.911749    2560 scope.go:117] "RemoveContainer" containerID="3d081088b657709ce193f096e1242fba133ba7bb0cecc9c7bab736a7bd3f2141"
Jul 24 21:31:32.912022 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:31:32.912001    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d081088b657709ce193f096e1242fba133ba7bb0cecc9c7bab736a7bd3f2141\": container with ID starting with 3d081088b657709ce193f096e1242fba133ba7bb0cecc9c7bab736a7bd3f2141 not found: ID does not exist" containerID="3d081088b657709ce193f096e1242fba133ba7bb0cecc9c7bab736a7bd3f2141"
Jul 24 21:31:32.912058 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.912033    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d081088b657709ce193f096e1242fba133ba7bb0cecc9c7bab736a7bd3f2141"} err="failed to get container status \"3d081088b657709ce193f096e1242fba133ba7bb0cecc9c7bab736a7bd3f2141\": rpc error: code = NotFound desc = could not find container \"3d081088b657709ce193f096e1242fba133ba7bb0cecc9c7bab736a7bd3f2141\": container with ID starting with 3d081088b657709ce193f096e1242fba133ba7bb0cecc9c7bab736a7bd3f2141 not found: ID does not exist"
Jul 24 21:31:32.912058 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.912054    2560 scope.go:117] "RemoveContainer" containerID="ba04f9641b0cdcef80657ae61b4586a4dca5bcc4dc006f98f4fda6dac6b5ea15"
Jul 24 21:31:32.912335 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:31:32.912317    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba04f9641b0cdcef80657ae61b4586a4dca5bcc4dc006f98f4fda6dac6b5ea15\": container with ID starting with ba04f9641b0cdcef80657ae61b4586a4dca5bcc4dc006f98f4fda6dac6b5ea15 not found: ID does not exist" containerID="ba04f9641b0cdcef80657ae61b4586a4dca5bcc4dc006f98f4fda6dac6b5ea15"
Jul 24 21:31:32.912386 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.912342    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba04f9641b0cdcef80657ae61b4586a4dca5bcc4dc006f98f4fda6dac6b5ea15"} err="failed to get container status \"ba04f9641b0cdcef80657ae61b4586a4dca5bcc4dc006f98f4fda6dac6b5ea15\": rpc error: code = NotFound desc = could not find container \"ba04f9641b0cdcef80657ae61b4586a4dca5bcc4dc006f98f4fda6dac6b5ea15\": container with ID starting with ba04f9641b0cdcef80657ae61b4586a4dca5bcc4dc006f98f4fda6dac6b5ea15 not found: ID does not exist"
Jul 24 21:31:32.933437 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.933418    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"]
Jul 24 21:31:32.947816 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.947797    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6bdca-predictor-fdff74f97-flc24"]
Jul 24 21:31:32.960309 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.960293    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:31:32.996697 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.996676    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23b050fc-7979-4bc1-998d-fda8656aab29-kube-rbac-proxy-sar-config\") pod \"23b050fc-7979-4bc1-998d-fda8656aab29\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") "
Jul 24 21:31:32.996794 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.996707    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b050fc-7979-4bc1-998d-fda8656aab29-proxy-tls\") pod \"23b050fc-7979-4bc1-998d-fda8656aab29\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") "
Jul 24 21:31:32.996794 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.996755    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h72v\" (UniqueName: \"kubernetes.io/projected/23b050fc-7979-4bc1-998d-fda8656aab29-kube-api-access-6h72v\") pod \"23b050fc-7979-4bc1-998d-fda8656aab29\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") "
Jul 24 21:31:32.996794 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.996792    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23b050fc-7979-4bc1-998d-fda8656aab29-cabundle-cert\") pod \"23b050fc-7979-4bc1-998d-fda8656aab29\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") "
Jul 24 21:31:32.996962 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.996827    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b050fc-7979-4bc1-998d-fda8656aab29-kserve-provision-location\") pod \"23b050fc-7979-4bc1-998d-fda8656aab29\" (UID: \"23b050fc-7979-4bc1-998d-fda8656aab29\") "
Jul 24 21:31:32.997181 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.997155    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b050fc-7979-4bc1-998d-fda8656aab29-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "23b050fc-7979-4bc1-998d-fda8656aab29" (UID: "23b050fc-7979-4bc1-998d-fda8656aab29"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:31:32.997280 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.997257    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b050fc-7979-4bc1-998d-fda8656aab29-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "23b050fc-7979-4bc1-998d-fda8656aab29" (UID: "23b050fc-7979-4bc1-998d-fda8656aab29"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:31:32.997353 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.997308    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b050fc-7979-4bc1-998d-fda8656aab29-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "23b050fc-7979-4bc1-998d-fda8656aab29" (UID: "23b050fc-7979-4bc1-998d-fda8656aab29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:31:32.998979 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.998959    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b050fc-7979-4bc1-998d-fda8656aab29-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "23b050fc-7979-4bc1-998d-fda8656aab29" (UID: "23b050fc-7979-4bc1-998d-fda8656aab29"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:31:32.999172 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:32.999148    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b050fc-7979-4bc1-998d-fda8656aab29-kube-api-access-6h72v" (OuterVolumeSpecName: "kube-api-access-6h72v") pod "23b050fc-7979-4bc1-998d-fda8656aab29" (UID: "23b050fc-7979-4bc1-998d-fda8656aab29"). InnerVolumeSpecName "kube-api-access-6h72v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:31:33.097767 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:33.097695    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h72v\" (UniqueName: \"kubernetes.io/projected/23b050fc-7979-4bc1-998d-fda8656aab29-kube-api-access-6h72v\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:31:33.097767 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:33.097720    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23b050fc-7979-4bc1-998d-fda8656aab29-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:31:33.097767 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:33.097733    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23b050fc-7979-4bc1-998d-fda8656aab29-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:31:33.097767 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:33.097746    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23b050fc-7979-4bc1-998d-fda8656aab29-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:31:33.097767 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:33.097758    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23b050fc-7979-4bc1-998d-fda8656aab29-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:31:33.754116 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:33.754074    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" path="/var/lib/kubelet/pods/dabc18d6-628b-4c48-9117-8bdb56c6bebd/volumes"
Jul 24 21:31:33.895941 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:33.895913    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"
Jul 24 21:31:33.895941 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:33.895918    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz" event={"ID":"23b050fc-7979-4bc1-998d-fda8656aab29","Type":"ContainerDied","Data":"a2a5c70b990625ca2d023f042a1e68376070f27306c6ec464c0507e3dcd3474d"}
Jul 24 21:31:33.896511 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:33.895995    2560 scope.go:117] "RemoveContainer" containerID="6a9d9fb60536f9c549f7cba4925ec3b6fe6a11bce658514c10d6ad49036d78b5"
Jul 24 21:31:33.904509 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:33.904492    2560 scope.go:117] "RemoveContainer" containerID="4ae35e7a8a6923ac9e75990e21fdbf22ba34f80208fe2ad233463f2ad9c63ea8"
Jul 24 21:31:33.911631 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:33.911607    2560 scope.go:117] "RemoveContainer" containerID="c3362cc0860f403b02149f89141af1a70e7f4bfa0a2ea2790bb5539de9373746"
Jul 24 21:31:33.913597 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:33.913577    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"]
Jul 24 21:31:33.917735 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:33.917716    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6bdca-predictor-7f94d7d44c-rm4kz"]
Jul 24 21:31:35.754006 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:35.753974    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" path="/var/lib/kubelet/pods/23b050fc-7979-4bc1-998d-fda8656aab29/volumes"
Jul 24 21:31:37.894285 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:37.894244    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:31:38.352661 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.352587    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:31:38.406289 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406253    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"]
Jul 24 21:31:38.406580 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406569    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="storage-initializer"
Jul 24 21:31:38.406627 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406582    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="storage-initializer"
Jul 24 21:31:38.406627 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406594    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="storage-initializer"
Jul 24 21:31:38.406627 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406599    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="storage-initializer"
Jul 24 21:31:38.406627 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406606    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kserve-container"
Jul 24 21:31:38.406627 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406611    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kserve-container"
Jul 24 21:31:38.406627 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406620    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kube-rbac-proxy"
Jul 24 21:31:38.406627 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406625    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kube-rbac-proxy"
Jul 24 21:31:38.406829 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406632    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kserve-container"
Jul 24 21:31:38.406829 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406638    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kserve-container"
Jul 24 21:31:38.406829 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406659    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kube-rbac-proxy"
Jul 24 21:31:38.406829 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406665    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kube-rbac-proxy"
Jul 24 21:31:38.406829 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406711    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kserve-container"
Jul 24 21:31:38.406829 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406720    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="23b050fc-7979-4bc1-998d-fda8656aab29" containerName="kube-rbac-proxy"
Jul 24 21:31:38.406829 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406725    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kube-rbac-proxy"
Jul 24 21:31:38.406829 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.406732    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dabc18d6-628b-4c48-9117-8bdb56c6bebd" containerName="kserve-container"
Jul 24 21:31:38.411624 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.411604    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.414016 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.413992    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-103c4-kube-rbac-proxy-sar-config\""
Jul 24 21:31:38.414175 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.414043    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:31:38.414175 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.414043    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:31:38.414175 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.414115    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-103c4-predictor-serving-cert\""
Jul 24 21:31:38.422027 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.422006    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"]
Jul 24 21:31:38.437311 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.437285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") " pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.437409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.437328    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-proxy-tls\") pod \"isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") " pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.437409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.437365    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kserve-provision-location\") pod \"isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") " pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.437409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.437391    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-cabundle-cert\") pod \"isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") " pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.437522 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.437458    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r92z6\" (UniqueName: \"kubernetes.io/projected/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kube-api-access-r92z6\") pod \"isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") " pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.444636 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.444599    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:31:38.537903 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.537870    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r92z6\" (UniqueName: \"kubernetes.io/projected/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kube-api-access-r92z6\") pod \"isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") " pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.537903 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.537912    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") " pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.538232 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.537971    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-proxy-tls\") pod \"isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") " pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.538232 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.538044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kserve-provision-location\") pod \"isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") " pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.538232 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.538084    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-cabundle-cert\") pod \"isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") " pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.538489 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.538461    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kserve-provision-location\") pod \"isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") " pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.538627 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.538597    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") " pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.538845 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.538698    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-cabundle-cert\") pod \"isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") " pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.540457 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.540437    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-proxy-tls\") pod \"isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") " pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.547046 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.547025    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r92z6\" (UniqueName: \"kubernetes.io/projected/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kube-api-access-r92z6\") pod \"isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") " pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.722493 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.722458    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:38.858371 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.858341    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"]
Jul 24 21:31:38.861566 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:31:38.861544    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8066dfd8_1b5a_4c71_ada7_4d679d4c6d28.slice/crio-6300a7ee21f7ff1fc4406baff5d7e47efc0e70cf6a71c47007f67a2570167372 WatchSource:0}: Error finding container 6300a7ee21f7ff1fc4406baff5d7e47efc0e70cf6a71c47007f67a2570167372: Status 404 returned error can't find the container with id 6300a7ee21f7ff1fc4406baff5d7e47efc0e70cf6a71c47007f67a2570167372
Jul 24 21:31:38.920251 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:38.920220    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" event={"ID":"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28","Type":"ContainerStarted","Data":"6300a7ee21f7ff1fc4406baff5d7e47efc0e70cf6a71c47007f67a2570167372"}
Jul 24 21:31:39.928034 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:39.928003    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" event={"ID":"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28","Type":"ContainerStarted","Data":"69bef447482d0c50da218276b9b027163f3623cb0dafac9764f67f030e396bf7"}
Jul 24 21:31:42.943698 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:42.943664    2560 generic.go:356] "Generic (PLEG): container finished" podID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerID="69bef447482d0c50da218276b9b027163f3623cb0dafac9764f67f030e396bf7" exitCode=0
Jul 24 21:31:42.944005 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:42.943737    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" event={"ID":"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28","Type":"ContainerDied","Data":"69bef447482d0c50da218276b9b027163f3623cb0dafac9764f67f030e396bf7"}
Jul 24 21:31:43.951420 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:43.951379    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" event={"ID":"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28","Type":"ContainerStarted","Data":"7eccf99b43c1ea66272d33fea28a31aada5bd3f52e95ddb2d976fc56f8b9c985"}
Jul 24 21:31:43.951893 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:43.951460    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" event={"ID":"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28","Type":"ContainerStarted","Data":"91d6b4cf3d9bdac2c036acc4861d3e3a379d05664d86b9d58266cd395e06250e"}
Jul 24 21:31:43.951893 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:43.951501    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" event={"ID":"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28","Type":"ContainerStarted","Data":"63c6d309883838b536f46603dad251ebe2cb21b1406566217a2fb9cf4ae11539"}
Jul 24 21:31:43.951893 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:43.951740    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:43.972643 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:43.972603    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podStartSLOduration=2.063137851 podStartE2EDuration="5.972588906s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.909451055s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:31:43.971845455 +0000 UTC m=+982.728191268" watchObservedRunningTime="2026-07-24 21:31:43.972588906 +0000 UTC m=+982.728934713"
Jul 24 21:31:44.955699 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:44.955666    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:44.956089 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:44.955746    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:44.957247 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:44.957210    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 24 21:31:44.957933 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:44.957907    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:31:45.960417 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:45.960358    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 24 21:31:45.960828 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:45.960764    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:31:50.964518 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:50.964489    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:31:50.965069 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:50.965030    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 24 21:31:50.965377 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:31:50.965353    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:32:00.965602 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:32:00.965562    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 24 21:32:00.966186 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:32:00.966160    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:32:10.965765 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:32:10.965723    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 24 21:32:10.966325 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:32:10.966296    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:32:20.965706 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:32:20.965661    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 24 21:32:20.966262 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:32:20.966095    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:32:30.965007 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:32:30.964960    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 24 21:32:30.965602 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:32:30.965577    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:32:40.965330 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:32:40.965290    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 24 21:32:40.965753 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:32:40.965727    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:32:50.964964 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:32:50.964920    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 24 21:32:50.965536 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:32:50.965499    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:32:52.749993 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:32:52.749942    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 24 21:32:52.750477 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:32:52.750379    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:33:02.750927 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:02.750895    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:33:02.751387 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:02.751001    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:33:13.422427 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.422393    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-103c4-predictor-86555d876f-r7wbl_8a58d115-6917-4419-b105-6e5498a2e959/kserve-container/0.log"
Jul 24 21:33:13.604887 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.604855    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"]
Jul 24 21:33:13.605253 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.605195    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container" containerID="cri-o://63c6d309883838b536f46603dad251ebe2cb21b1406566217a2fb9cf4ae11539" gracePeriod=30
Jul 24 21:33:13.605352 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.605265    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent" containerID="cri-o://7eccf99b43c1ea66272d33fea28a31aada5bd3f52e95ddb2d976fc56f8b9c985" gracePeriod=30
Jul 24 21:33:13.605352 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.605282    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kube-rbac-proxy" containerID="cri-o://91d6b4cf3d9bdac2c036acc4861d3e3a379d05664d86b9d58266cd395e06250e" gracePeriod=30
Jul 24 21:33:13.647714 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.647678    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:33:13.712145 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.712016    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"]
Jul 24 21:33:13.712971 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.712490    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl" podUID="8a58d115-6917-4419-b105-6e5498a2e959" containerName="kube-rbac-proxy" containerID="cri-o://6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd" gracePeriod=30
Jul 24 21:33:13.712971 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.712642    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl" podUID="8a58d115-6917-4419-b105-6e5498a2e959" containerName="kserve-container" containerID="cri-o://28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3" gracePeriod=30
Jul 24 21:33:13.714447 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.714417    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:33:13.766647 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.766616    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"]
Jul 24 21:33:13.770147 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.770107    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:13.772297 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.772277    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-6f3c9-predictor-serving-cert\""
Jul 24 21:33:13.772413 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.772278    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-6f3c9-kube-rbac-proxy-sar-config\""
Jul 24 21:33:13.780370 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.780345    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:33:13.780533 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.780516    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"]
Jul 24 21:33:13.924298 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.924269    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c949ede-2748-4721-b5d0-ae6d607d85ba-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:13.924460 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.924306    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c949ede-2748-4721-b5d0-ae6d607d85ba-proxy-tls\") pod \"isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:13.924460 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.924330    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c949ede-2748-4721-b5d0-ae6d607d85ba-cabundle-cert\") pod \"isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:13.924460 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.924447    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c949ede-2748-4721-b5d0-ae6d607d85ba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:13.924595 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.924492    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q59rs\" (UniqueName: \"kubernetes.io/projected/3c949ede-2748-4721-b5d0-ae6d607d85ba-kube-api-access-q59rs\") pod \"isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:13.962720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.962662    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:33:13.997110 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:13.997079    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:33:14.025892 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.025862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c949ede-2748-4721-b5d0-ae6d607d85ba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:14.026053 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.025911    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q59rs\" (UniqueName: \"kubernetes.io/projected/3c949ede-2748-4721-b5d0-ae6d607d85ba-kube-api-access-q59rs\") pod \"isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:14.026053 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.025950    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c949ede-2748-4721-b5d0-ae6d607d85ba-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:14.026053 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.025970    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c949ede-2748-4721-b5d0-ae6d607d85ba-proxy-tls\") pod \"isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:14.026053 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.025989    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c949ede-2748-4721-b5d0-ae6d607d85ba-cabundle-cert\") pod \"isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:14.026423 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.026398    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c949ede-2748-4721-b5d0-ae6d607d85ba-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:14.026629 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.026613    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c949ede-2748-4721-b5d0-ae6d607d85ba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:14.026673 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.026651    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c949ede-2748-4721-b5d0-ae6d607d85ba-cabundle-cert\") pod \"isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:14.028499 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.028483    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c949ede-2748-4721-b5d0-ae6d607d85ba-proxy-tls\") pod \"isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:14.034680 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.034654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q59rs\" (UniqueName: \"kubernetes.io/projected/3c949ede-2748-4721-b5d0-ae6d607d85ba-kube-api-access-q59rs\") pod \"isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:14.120017 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.119986    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:14.129643 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.129619    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m54vg\" (UniqueName: \"kubernetes.io/projected/8a58d115-6917-4419-b105-6e5498a2e959-kube-api-access-m54vg\") pod \"8a58d115-6917-4419-b105-6e5498a2e959\" (UID: \"8a58d115-6917-4419-b105-6e5498a2e959\") "
Jul 24 21:33:14.129766 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.129690    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a58d115-6917-4419-b105-6e5498a2e959-proxy-tls\") pod \"8a58d115-6917-4419-b105-6e5498a2e959\" (UID: \"8a58d115-6917-4419-b105-6e5498a2e959\") "
Jul 24 21:33:14.129766 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.129745    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a58d115-6917-4419-b105-6e5498a2e959-kube-rbac-proxy-sar-config\") pod \"8a58d115-6917-4419-b105-6e5498a2e959\" (UID: \"8a58d115-6917-4419-b105-6e5498a2e959\") "
Jul 24 21:33:14.130184 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.130155    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a58d115-6917-4419-b105-6e5498a2e959-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a58d115-6917-4419-b105-6e5498a2e959" (UID: "8a58d115-6917-4419-b105-6e5498a2e959"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:33:14.131896 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.131873    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a58d115-6917-4419-b105-6e5498a2e959-kube-api-access-m54vg" (OuterVolumeSpecName: "kube-api-access-m54vg") pod "8a58d115-6917-4419-b105-6e5498a2e959" (UID: "8a58d115-6917-4419-b105-6e5498a2e959"). InnerVolumeSpecName "kube-api-access-m54vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:33:14.131999 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.131909    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a58d115-6917-4419-b105-6e5498a2e959-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a58d115-6917-4419-b105-6e5498a2e959" (UID: "8a58d115-6917-4419-b105-6e5498a2e959"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:33:14.230958 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.230859    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m54vg\" (UniqueName: \"kubernetes.io/projected/8a58d115-6917-4419-b105-6e5498a2e959-kube-api-access-m54vg\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:33:14.230958 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.230888    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a58d115-6917-4419-b105-6e5498a2e959-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:33:14.230958 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.230897    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a58d115-6917-4419-b105-6e5498a2e959-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:33:14.253847 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.253825    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"]
Jul 24 21:33:14.256302 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:33:14.256275    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c949ede_2748_4721_b5d0_ae6d607d85ba.slice/crio-087b151e75f4ba61a2755ca445004dc2d7d957e2b7542a0b80465b6b40fda0ca WatchSource:0}: Error finding container 087b151e75f4ba61a2755ca445004dc2d7d957e2b7542a0b80465b6b40fda0ca: Status 404 returned error can't find the container with id 087b151e75f4ba61a2755ca445004dc2d7d957e2b7542a0b80465b6b40fda0ca
Jul 24 21:33:14.321105 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.321083    2560 generic.go:356] "Generic (PLEG): container finished" podID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerID="91d6b4cf3d9bdac2c036acc4861d3e3a379d05664d86b9d58266cd395e06250e" exitCode=2
Jul 24 21:33:14.321233 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.321112    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" event={"ID":"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28","Type":"ContainerDied","Data":"91d6b4cf3d9bdac2c036acc4861d3e3a379d05664d86b9d58266cd395e06250e"}
Jul 24 21:33:14.322661 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.322637    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" event={"ID":"3c949ede-2748-4721-b5d0-ae6d607d85ba","Type":"ContainerStarted","Data":"087b151e75f4ba61a2755ca445004dc2d7d957e2b7542a0b80465b6b40fda0ca"}
Jul 24 21:33:14.324254 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.324235    2560 generic.go:356] "Generic (PLEG): container finished" podID="8a58d115-6917-4419-b105-6e5498a2e959" containerID="6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd" exitCode=2
Jul 24 21:33:14.324254 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.324253    2560 generic.go:356] "Generic (PLEG): container finished" podID="8a58d115-6917-4419-b105-6e5498a2e959" containerID="28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3" exitCode=2
Jul 24 21:33:14.324393 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.324276    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl" event={"ID":"8a58d115-6917-4419-b105-6e5498a2e959","Type":"ContainerDied","Data":"6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd"}
Jul 24 21:33:14.324393 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.324281    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"
Jul 24 21:33:14.324393 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.324320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl" event={"ID":"8a58d115-6917-4419-b105-6e5498a2e959","Type":"ContainerDied","Data":"28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3"}
Jul 24 21:33:14.324393 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.324342    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl" event={"ID":"8a58d115-6917-4419-b105-6e5498a2e959","Type":"ContainerDied","Data":"486cb23a6467e3c2b547220a0f799dd9c5c5c473cca6dbf9d089d20567ef54e8"}
Jul 24 21:33:14.324393 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.324367    2560 scope.go:117] "RemoveContainer" containerID="6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd"
Jul 24 21:33:14.333423 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.333407    2560 scope.go:117] "RemoveContainer" containerID="28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3"
Jul 24 21:33:14.342113 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.342090    2560 scope.go:117] "RemoveContainer" containerID="6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd"
Jul 24 21:33:14.342468 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:33:14.342446    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd\": container with ID starting with 6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd not found: ID does not exist" containerID="6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd"
Jul 24 21:33:14.342533 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.342478    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd"} err="failed to get container status \"6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd\": rpc error: code = NotFound desc = could not find container \"6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd\": container with ID starting with 6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd not found: ID does not exist"
Jul 24 21:33:14.342533 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.342495    2560 scope.go:117] "RemoveContainer" containerID="28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3"
Jul 24 21:33:14.342739 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:33:14.342723    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3\": container with ID starting with 28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3 not found: ID does not exist" containerID="28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3"
Jul 24 21:33:14.342784 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.342744    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3"} err="failed to get container status \"28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3\": rpc error: code = NotFound desc = could not find container \"28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3\": container with ID starting with 28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3 not found: ID does not exist"
Jul 24 21:33:14.342784 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.342758    2560 scope.go:117] "RemoveContainer" containerID="6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd"
Jul 24 21:33:14.343050 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.343025    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd"} err="failed to get container status \"6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd\": rpc error: code = NotFound desc = could not find container \"6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd\": container with ID starting with 6565c6d83340ee9e410a07c5710c9a5009f7305742266dfbf7ece1dfa7ba55fd not found: ID does not exist"
Jul 24 21:33:14.343110 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.343054    2560 scope.go:117] "RemoveContainer" containerID="28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3"
Jul 24 21:33:14.343351 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.343332    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3"} err="failed to get container status \"28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3\": rpc error: code = NotFound desc = could not find container \"28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3\": container with ID starting with 28dda9e6f96fd0e17331f68276b859a81780de1d7c07c9983af0f8c56b4fd1a3 not found: ID does not exist"
Jul 24 21:33:14.352719 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.352671    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"]
Jul 24 21:33:14.354889 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:14.354871    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-103c4-predictor-86555d876f-r7wbl"]
Jul 24 21:33:15.330306 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:15.330270    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" event={"ID":"3c949ede-2748-4721-b5d0-ae6d607d85ba","Type":"ContainerStarted","Data":"ce2a4dc7268ca267c843d9a0737591423ddb679225f411057042e7ed8bc4a2ce"}
Jul 24 21:33:15.753827 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:15.753792    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a58d115-6917-4419-b105-6e5498a2e959" path="/var/lib/kubelet/pods/8a58d115-6917-4419-b105-6e5498a2e959/volumes"
Jul 24 21:33:15.961196 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:15.961154    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 24 21:33:18.345377 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:18.345352    2560 generic.go:356] "Generic (PLEG): container finished" podID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerID="63c6d309883838b536f46603dad251ebe2cb21b1406566217a2fb9cf4ae11539" exitCode=0
Jul 24 21:33:18.345709 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:18.345420    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" event={"ID":"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28","Type":"ContainerDied","Data":"63c6d309883838b536f46603dad251ebe2cb21b1406566217a2fb9cf4ae11539"}
Jul 24 21:33:18.346825 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:18.346806    2560 generic.go:356] "Generic (PLEG): container finished" podID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerID="ce2a4dc7268ca267c843d9a0737591423ddb679225f411057042e7ed8bc4a2ce" exitCode=0
Jul 24 21:33:18.346929 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:18.346882    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" event={"ID":"3c949ede-2748-4721-b5d0-ae6d607d85ba","Type":"ContainerDied","Data":"ce2a4dc7268ca267c843d9a0737591423ddb679225f411057042e7ed8bc4a2ce"}
Jul 24 21:33:19.352438 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:19.352410    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" event={"ID":"3c949ede-2748-4721-b5d0-ae6d607d85ba","Type":"ContainerStarted","Data":"187e12cce7e5dc07a38d60f4cd067c91a697f22c12b2b9769126b45985fe3653"}
Jul 24 21:33:19.352832 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:19.352476    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" event={"ID":"3c949ede-2748-4721-b5d0-ae6d607d85ba","Type":"ContainerStarted","Data":"3ec1d13a2a1176bc2b6d0ae33db5b10863b84a6f14e0d7cac9553456f5a001bf"}
Jul 24 21:33:19.352832 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:19.352752    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:19.352928 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:19.352910    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:19.354216 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:19.354189    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:33:19.383244 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:19.383202    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podStartSLOduration=2.462634515 podStartE2EDuration="6.383188176s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.920553661s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:33:19.382255284 +0000 UTC m=+1078.138601097" watchObservedRunningTime="2026-07-24 21:33:19.383188176 +0000 UTC m=+1078.139533988"
Jul 24 21:33:20.356629 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:20.356587    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:33:20.960728 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:20.960688    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 24 21:33:22.749890 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:22.749848    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 24 21:33:22.750339 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:22.750177    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:33:25.361026 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:25.360998    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:33:25.361487 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:25.361464    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:33:25.960763 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:25.960720    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 24 21:33:25.960917 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:25.960852    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:33:30.960667 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:30.960625    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 24 21:33:32.750031 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:32.749982    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 24 21:33:32.750455 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:32.750325    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:33:35.361705 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:35.361665    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:33:35.961481 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:35.961443    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 24 21:33:40.961092 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:40.961052    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 24 21:33:42.750064 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:42.750015    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 24 21:33:42.750533 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:42.750196    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:33:42.750533 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:42.750439    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 21:33:42.750656 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:42.750541    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:33:43.760959 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.760936    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:33:43.855233 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.855209    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kserve-provision-location\") pod \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") "
Jul 24 21:33:43.855397 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.855243    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-cabundle-cert\") pod \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") "
Jul 24 21:33:43.855397 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.855291    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-proxy-tls\") pod \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") "
Jul 24 21:33:43.855397 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.855314    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r92z6\" (UniqueName: \"kubernetes.io/projected/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kube-api-access-r92z6\") pod \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") "
Jul 24 21:33:43.855397 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.855361    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kube-rbac-proxy-sar-config\") pod \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\" (UID: \"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28\") "
Jul 24 21:33:43.855624 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.855494    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" (UID: "8066dfd8-1b5a-4c71-ada7-4d679d4c6d28"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:33:43.855691 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.855650    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" (UID: "8066dfd8-1b5a-4c71-ada7-4d679d4c6d28"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:33:43.855813 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.855781    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" (UID: "8066dfd8-1b5a-4c71-ada7-4d679d4c6d28"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:33:43.857649 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.857622    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kube-api-access-r92z6" (OuterVolumeSpecName: "kube-api-access-r92z6") pod "8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" (UID: "8066dfd8-1b5a-4c71-ada7-4d679d4c6d28"). InnerVolumeSpecName "kube-api-access-r92z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:33:43.857649 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.857634    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" (UID: "8066dfd8-1b5a-4c71-ada7-4d679d4c6d28"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:33:43.956192 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.956161    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:33:43.956192 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.956189    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:33:43.956380 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.956204    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:33:43.956380 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.956215    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r92z6\" (UniqueName: \"kubernetes.io/projected/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kube-api-access-r92z6\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:33:43.956380 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:43.956227    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:33:44.452533 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.452496    2560 generic.go:356] "Generic (PLEG): container finished" podID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerID="7eccf99b43c1ea66272d33fea28a31aada5bd3f52e95ddb2d976fc56f8b9c985" exitCode=0
Jul 24 21:33:44.452704 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.452564    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"
Jul 24 21:33:44.452704 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.452575    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" event={"ID":"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28","Type":"ContainerDied","Data":"7eccf99b43c1ea66272d33fea28a31aada5bd3f52e95ddb2d976fc56f8b9c985"}
Jul 24 21:33:44.452704 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.452640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr" event={"ID":"8066dfd8-1b5a-4c71-ada7-4d679d4c6d28","Type":"ContainerDied","Data":"6300a7ee21f7ff1fc4406baff5d7e47efc0e70cf6a71c47007f67a2570167372"}
Jul 24 21:33:44.452704 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.452673    2560 scope.go:117] "RemoveContainer" containerID="7eccf99b43c1ea66272d33fea28a31aada5bd3f52e95ddb2d976fc56f8b9c985"
Jul 24 21:33:44.461874 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.461857    2560 scope.go:117] "RemoveContainer" containerID="91d6b4cf3d9bdac2c036acc4861d3e3a379d05664d86b9d58266cd395e06250e"
Jul 24 21:33:44.469908 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.469886    2560 scope.go:117] "RemoveContainer" containerID="63c6d309883838b536f46603dad251ebe2cb21b1406566217a2fb9cf4ae11539"
Jul 24 21:33:44.474656 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.474636    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"]
Jul 24 21:33:44.478344 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.478327    2560 scope.go:117] "RemoveContainer" containerID="69bef447482d0c50da218276b9b027163f3623cb0dafac9764f67f030e396bf7"
Jul 24 21:33:44.479274 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.479261    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-103c4-predictor-95c8bff79-lgvxr"]
Jul 24 21:33:44.485536 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.485521    2560 scope.go:117] "RemoveContainer" containerID="7eccf99b43c1ea66272d33fea28a31aada5bd3f52e95ddb2d976fc56f8b9c985"
Jul 24 21:33:44.485777 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:33:44.485756    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eccf99b43c1ea66272d33fea28a31aada5bd3f52e95ddb2d976fc56f8b9c985\": container with ID starting with 7eccf99b43c1ea66272d33fea28a31aada5bd3f52e95ddb2d976fc56f8b9c985 not found: ID does not exist" containerID="7eccf99b43c1ea66272d33fea28a31aada5bd3f52e95ddb2d976fc56f8b9c985"
Jul 24 21:33:44.485844 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.485783    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eccf99b43c1ea66272d33fea28a31aada5bd3f52e95ddb2d976fc56f8b9c985"} err="failed to get container status \"7eccf99b43c1ea66272d33fea28a31aada5bd3f52e95ddb2d976fc56f8b9c985\": rpc error: code = NotFound desc = could not find container \"7eccf99b43c1ea66272d33fea28a31aada5bd3f52e95ddb2d976fc56f8b9c985\": container with ID starting with 7eccf99b43c1ea66272d33fea28a31aada5bd3f52e95ddb2d976fc56f8b9c985 not found: ID does not exist"
Jul 24 21:33:44.485844 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.485800    2560 scope.go:117] "RemoveContainer" containerID="91d6b4cf3d9bdac2c036acc4861d3e3a379d05664d86b9d58266cd395e06250e"
Jul 24 21:33:44.486041 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:33:44.486023    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d6b4cf3d9bdac2c036acc4861d3e3a379d05664d86b9d58266cd395e06250e\": container with ID starting with 91d6b4cf3d9bdac2c036acc4861d3e3a379d05664d86b9d58266cd395e06250e not found: ID does not exist" containerID="91d6b4cf3d9bdac2c036acc4861d3e3a379d05664d86b9d58266cd395e06250e"
Jul 24 21:33:44.486082 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.486049    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d6b4cf3d9bdac2c036acc4861d3e3a379d05664d86b9d58266cd395e06250e"} err="failed to get container status \"91d6b4cf3d9bdac2c036acc4861d3e3a379d05664d86b9d58266cd395e06250e\": rpc error: code = NotFound desc = could not find container \"91d6b4cf3d9bdac2c036acc4861d3e3a379d05664d86b9d58266cd395e06250e\": container with ID starting with 91d6b4cf3d9bdac2c036acc4861d3e3a379d05664d86b9d58266cd395e06250e not found: ID does not exist"
Jul 24 21:33:44.486082 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.486064    2560 scope.go:117] "RemoveContainer" containerID="63c6d309883838b536f46603dad251ebe2cb21b1406566217a2fb9cf4ae11539"
Jul 24 21:33:44.486322 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:33:44.486305    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c6d309883838b536f46603dad251ebe2cb21b1406566217a2fb9cf4ae11539\": container with ID starting with 63c6d309883838b536f46603dad251ebe2cb21b1406566217a2fb9cf4ae11539 not found: ID does not exist" containerID="63c6d309883838b536f46603dad251ebe2cb21b1406566217a2fb9cf4ae11539"
Jul 24 21:33:44.486378 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.486325    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c6d309883838b536f46603dad251ebe2cb21b1406566217a2fb9cf4ae11539"} err="failed to get container status \"63c6d309883838b536f46603dad251ebe2cb21b1406566217a2fb9cf4ae11539\": rpc error: code = NotFound desc = could not find container \"63c6d309883838b536f46603dad251ebe2cb21b1406566217a2fb9cf4ae11539\": container with ID starting with 63c6d309883838b536f46603dad251ebe2cb21b1406566217a2fb9cf4ae11539 not found: ID does not exist"
Jul 24 21:33:44.486378 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.486338    2560 scope.go:117] "RemoveContainer" containerID="69bef447482d0c50da218276b9b027163f3623cb0dafac9764f67f030e396bf7"
Jul 24 21:33:44.486577 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:33:44.486562    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69bef447482d0c50da218276b9b027163f3623cb0dafac9764f67f030e396bf7\": container with ID starting with 69bef447482d0c50da218276b9b027163f3623cb0dafac9764f67f030e396bf7 not found: ID does not exist" containerID="69bef447482d0c50da218276b9b027163f3623cb0dafac9764f67f030e396bf7"
Jul 24 21:33:44.486619 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:44.486582    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69bef447482d0c50da218276b9b027163f3623cb0dafac9764f67f030e396bf7"} err="failed to get container status \"69bef447482d0c50da218276b9b027163f3623cb0dafac9764f67f030e396bf7\": rpc error: code = NotFound desc = could not find container \"69bef447482d0c50da218276b9b027163f3623cb0dafac9764f67f030e396bf7\": container with ID starting with 69bef447482d0c50da218276b9b027163f3623cb0dafac9764f67f030e396bf7 not found: ID does not exist"
Jul 24 21:33:45.362071 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:45.362030    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:33:45.758661 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:45.758557    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" path="/var/lib/kubelet/pods/8066dfd8-1b5a-4c71-ada7-4d679d4c6d28/volumes"
Jul 24 21:33:55.361429 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:33:55.361392    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:34:05.361370 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:34:05.361329    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:34:15.361447 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:34:15.361399    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:34:21.085789 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:34:21.085756    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:34:21.994792 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:34:21.994697    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:34:22.049693 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:34:22.049663    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:34:25.362145 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:34:25.362091    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:34:35.362028 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:34:35.361979    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:34:45.361818 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:34:45.361730    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:34:49.749358 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:34:49.749313    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:34:59.754117 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:34:59.754067    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:35:09.750258 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:09.750219    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:35:19.749322 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:19.749274    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:35:21.695306 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:21.695276    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:35:21.700840 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:21.700823    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:35:29.749501 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:29.749450    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:35:39.750312 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:39.750263    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:35:41.836333 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:41.836298    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:35:45.250549 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:45.250513    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:35:49.758873 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:49.758845    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:35:53.894361 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:53.894328    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"]
Jul 24 21:35:53.894885 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:53.894584    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" containerID="cri-o://3ec1d13a2a1176bc2b6d0ae33db5b10863b84a6f14e0d7cac9553456f5a001bf" gracePeriod=30
Jul 24 21:35:53.894885 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:53.894598    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kube-rbac-proxy" containerID="cri-o://187e12cce7e5dc07a38d60f4cd067c91a697f22c12b2b9769126b45985fe3653" gracePeriod=30
Jul 24 21:35:53.914325 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:53.914296    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:35:54.971449 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:54.971416    2560 generic.go:356] "Generic (PLEG): container finished" podID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerID="187e12cce7e5dc07a38d60f4cd067c91a697f22c12b2b9769126b45985fe3653" exitCode=2
Jul 24 21:35:54.971837 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:54.971493    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" event={"ID":"3c949ede-2748-4721-b5d0-ae6d607d85ba","Type":"ContainerDied","Data":"187e12cce7e5dc07a38d60f4cd067c91a697f22c12b2b9769126b45985fe3653"}
Jul 24 21:35:55.357679 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:55.357564    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 24 21:35:59.043965 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.043926    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:35:59.104884 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.104849    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:35:59.108447 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108418    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"]
Jul 24 21:35:59.108722 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108711    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container"
Jul 24 21:35:59.108767 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108739    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container"
Jul 24 21:35:59.108767 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108748    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kube-rbac-proxy"
Jul 24 21:35:59.108767 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108754    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kube-rbac-proxy"
Jul 24 21:35:59.108767 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108760    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent"
Jul 24 21:35:59.108767 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108765    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent"
Jul 24 21:35:59.108916 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108775    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a58d115-6917-4419-b105-6e5498a2e959" containerName="kserve-container"
Jul 24 21:35:59.108916 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108780    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a58d115-6917-4419-b105-6e5498a2e959" containerName="kserve-container"
Jul 24 21:35:59.108916 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108786    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a58d115-6917-4419-b105-6e5498a2e959" containerName="kube-rbac-proxy"
Jul 24 21:35:59.108916 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108792    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a58d115-6917-4419-b105-6e5498a2e959" containerName="kube-rbac-proxy"
Jul 24 21:35:59.108916 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108802    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="storage-initializer"
Jul 24 21:35:59.108916 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108807    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="storage-initializer"
Jul 24 21:35:59.108916 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108863    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a58d115-6917-4419-b105-6e5498a2e959" containerName="kserve-container"
Jul 24 21:35:59.108916 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108871    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kube-rbac-proxy"
Jul 24 21:35:59.108916 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108877    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="kserve-container"
Jul 24 21:35:59.108916 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108882    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8066dfd8-1b5a-4c71-ada7-4d679d4c6d28" containerName="agent"
Jul 24 21:35:59.108916 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.108887    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a58d115-6917-4419-b105-6e5498a2e959" containerName="kube-rbac-proxy"
Jul 24 21:35:59.111877 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.111859    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.114173 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.114145    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 24 21:35:59.114277 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.114261    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 24 21:35:59.120540 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.120517    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"]
Jul 24 21:35:59.123986 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.123970    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.138659 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.138643    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 24 21:35:59.139788 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.139770    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"]
Jul 24 21:35:59.151992 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.151966    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"]
Jul 24 21:35:59.185613 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.185586    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:35:59.287944 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.287915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0748c7ef-2143-4a24-8c58-44c360b6f9f1-cabundle-cert\") pod \"isvc-canary-create-predictor-7f74cfd478-4vq6v\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.288095 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.287950    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5gc5\" (UniqueName: \"kubernetes.io/projected/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kube-api-access-k5gc5\") pod \"isvc-canary-create-predictor-7f74cfd478-4vq6v\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.288095 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.287970    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55w5b\" (UniqueName: \"kubernetes.io/projected/a5660e06-3b68-4753-a86e-9b09d40dc645-kube-api-access-55w5b\") pod \"isvc-canary-create-v2-predictor-75c7994655-5v8h2\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.288095 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.287993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kserve-provision-location\") pod \"isvc-canary-create-predictor-7f74cfd478-4vq6v\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.288095 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.288039    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0748c7ef-2143-4a24-8c58-44c360b6f9f1-proxy-tls\") pod \"isvc-canary-create-predictor-7f74cfd478-4vq6v\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.288095 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.288073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5660e06-3b68-4753-a86e-9b09d40dc645-proxy-tls\") pod \"isvc-canary-create-v2-predictor-75c7994655-5v8h2\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.288293 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.288101    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5660e06-3b68-4753-a86e-9b09d40dc645-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-75c7994655-5v8h2\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.288293 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.288152    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5660e06-3b68-4753-a86e-9b09d40dc645-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-75c7994655-5v8h2\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.288293 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.288185    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7f74cfd478-4vq6v\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.288293 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.288213    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5660e06-3b68-4753-a86e-9b09d40dc645-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-75c7994655-5v8h2\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.293109 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.293082    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:35:59.389166 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.389141    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0748c7ef-2143-4a24-8c58-44c360b6f9f1-cabundle-cert\") pod \"isvc-canary-create-predictor-7f74cfd478-4vq6v\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.389326 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.389181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5gc5\" (UniqueName: \"kubernetes.io/projected/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kube-api-access-k5gc5\") pod \"isvc-canary-create-predictor-7f74cfd478-4vq6v\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.389326 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.389203    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55w5b\" (UniqueName: \"kubernetes.io/projected/a5660e06-3b68-4753-a86e-9b09d40dc645-kube-api-access-55w5b\") pod \"isvc-canary-create-v2-predictor-75c7994655-5v8h2\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.389326 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.389233    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kserve-provision-location\") pod \"isvc-canary-create-predictor-7f74cfd478-4vq6v\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.389326 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.389273    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0748c7ef-2143-4a24-8c58-44c360b6f9f1-proxy-tls\") pod \"isvc-canary-create-predictor-7f74cfd478-4vq6v\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.389326 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.389300    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5660e06-3b68-4753-a86e-9b09d40dc645-proxy-tls\") pod \"isvc-canary-create-v2-predictor-75c7994655-5v8h2\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.389580 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.389328    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5660e06-3b68-4753-a86e-9b09d40dc645-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-75c7994655-5v8h2\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.389580 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.389362    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5660e06-3b68-4753-a86e-9b09d40dc645-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-75c7994655-5v8h2\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.389694 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.389609    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7f74cfd478-4vq6v\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.389694 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.389667    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5660e06-3b68-4753-a86e-9b09d40dc645-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-75c7994655-5v8h2\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.389811 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.389793    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kserve-provision-location\") pod \"isvc-canary-create-predictor-7f74cfd478-4vq6v\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.389881 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.389813    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5660e06-3b68-4753-a86e-9b09d40dc645-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-75c7994655-5v8h2\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.390587 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.390009    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0748c7ef-2143-4a24-8c58-44c360b6f9f1-cabundle-cert\") pod \"isvc-canary-create-predictor-7f74cfd478-4vq6v\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.390587 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.390300    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5660e06-3b68-4753-a86e-9b09d40dc645-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-75c7994655-5v8h2\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.390587 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.390340    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7f74cfd478-4vq6v\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.390587 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.390397    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5660e06-3b68-4753-a86e-9b09d40dc645-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-75c7994655-5v8h2\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.391951 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.391930    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0748c7ef-2143-4a24-8c58-44c360b6f9f1-proxy-tls\") pod \"isvc-canary-create-predictor-7f74cfd478-4vq6v\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.392296 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.392279    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5660e06-3b68-4753-a86e-9b09d40dc645-proxy-tls\") pod \"isvc-canary-create-v2-predictor-75c7994655-5v8h2\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.397420 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.397398    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5gc5\" (UniqueName: \"kubernetes.io/projected/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kube-api-access-k5gc5\") pod \"isvc-canary-create-predictor-7f74cfd478-4vq6v\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.397515 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.397400    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55w5b\" (UniqueName: \"kubernetes.io/projected/a5660e06-3b68-4753-a86e-9b09d40dc645-kube-api-access-55w5b\") pod \"isvc-canary-create-v2-predictor-75c7994655-5v8h2\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.422288 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.422257    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:35:59.433630 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.433481    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:35:59.566919 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.566885    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"]
Jul 24 21:35:59.571034 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:35:59.571009    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0748c7ef_2143_4a24_8c58_44c360b6f9f1.slice/crio-77b1ab11bf260fe498396b8b294a22e24b940bb788a39dd41d7a2faf8a6395a5 WatchSource:0}: Error finding container 77b1ab11bf260fe498396b8b294a22e24b940bb788a39dd41d7a2faf8a6395a5: Status 404 returned error can't find the container with id 77b1ab11bf260fe498396b8b294a22e24b940bb788a39dd41d7a2faf8a6395a5
Jul 24 21:35:59.593635 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.593612    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"]
Jul 24 21:35:59.601215 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:35:59.601188    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5660e06_3b68_4753_a86e_9b09d40dc645.slice/crio-1836e90c02703e25f9f93b3732d4525bc187852bf65a78b19504ec599f8ddd75 WatchSource:0}: Error finding container 1836e90c02703e25f9f93b3732d4525bc187852bf65a78b19504ec599f8ddd75: Status 404 returned error can't find the container with id 1836e90c02703e25f9f93b3732d4525bc187852bf65a78b19504ec599f8ddd75
Jul 24 21:35:59.750195 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.750151    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 24 21:35:59.993421 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.993321    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" event={"ID":"0748c7ef-2143-4a24-8c58-44c360b6f9f1","Type":"ContainerStarted","Data":"fcdb0a55e803e53bba675fd078e7067d7c01db7f4aa5c84f11f4a74b7e68982a"}
Jul 24 21:35:59.993421 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.993391    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" event={"ID":"0748c7ef-2143-4a24-8c58-44c360b6f9f1","Type":"ContainerStarted","Data":"77b1ab11bf260fe498396b8b294a22e24b940bb788a39dd41d7a2faf8a6395a5"}
Jul 24 21:35:59.995062 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.995034    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" event={"ID":"a5660e06-3b68-4753-a86e-9b09d40dc645","Type":"ContainerStarted","Data":"ee3ac5867d031975c90481082111a4cea2d96a40037a46365eb610a6a77ab82b"}
Jul 24 21:35:59.995228 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:35:59.995099    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" event={"ID":"a5660e06-3b68-4753-a86e-9b09d40dc645","Type":"ContainerStarted","Data":"1836e90c02703e25f9f93b3732d4525bc187852bf65a78b19504ec599f8ddd75"}
Jul 24 21:36:00.357413 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:00.357321    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jul 24 21:36:03.647436 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.647407    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:36:03.733847 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.733777    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c949ede-2748-4721-b5d0-ae6d607d85ba-kserve-provision-location\") pod \"3c949ede-2748-4721-b5d0-ae6d607d85ba\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") "
Jul 24 21:36:03.733847 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.733808    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c949ede-2748-4721-b5d0-ae6d607d85ba-cabundle-cert\") pod \"3c949ede-2748-4721-b5d0-ae6d607d85ba\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") "
Jul 24 21:36:03.733847 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.733838    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q59rs\" (UniqueName: \"kubernetes.io/projected/3c949ede-2748-4721-b5d0-ae6d607d85ba-kube-api-access-q59rs\") pod \"3c949ede-2748-4721-b5d0-ae6d607d85ba\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") "
Jul 24 21:36:03.734082 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.733913    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c949ede-2748-4721-b5d0-ae6d607d85ba-kube-rbac-proxy-sar-config\") pod \"3c949ede-2748-4721-b5d0-ae6d607d85ba\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") "
Jul 24 21:36:03.734082 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.733947    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c949ede-2748-4721-b5d0-ae6d607d85ba-proxy-tls\") pod \"3c949ede-2748-4721-b5d0-ae6d607d85ba\" (UID: \"3c949ede-2748-4721-b5d0-ae6d607d85ba\") "
Jul 24 21:36:03.734082 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.734054    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c949ede-2748-4721-b5d0-ae6d607d85ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c949ede-2748-4721-b5d0-ae6d607d85ba" (UID: "3c949ede-2748-4721-b5d0-ae6d607d85ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:36:03.734297 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.734215    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c949ede-2748-4721-b5d0-ae6d607d85ba-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:36:03.734297 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.734239    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c949ede-2748-4721-b5d0-ae6d607d85ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3c949ede-2748-4721-b5d0-ae6d607d85ba" (UID: "3c949ede-2748-4721-b5d0-ae6d607d85ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:36:03.734451 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.734430    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c949ede-2748-4721-b5d0-ae6d607d85ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3c949ede-2748-4721-b5d0-ae6d607d85ba" (UID: "3c949ede-2748-4721-b5d0-ae6d607d85ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:36:03.736121 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.736103    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c949ede-2748-4721-b5d0-ae6d607d85ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3c949ede-2748-4721-b5d0-ae6d607d85ba" (UID: "3c949ede-2748-4721-b5d0-ae6d607d85ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:36:03.736222 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.736194    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c949ede-2748-4721-b5d0-ae6d607d85ba-kube-api-access-q59rs" (OuterVolumeSpecName: "kube-api-access-q59rs") pod "3c949ede-2748-4721-b5d0-ae6d607d85ba" (UID: "3c949ede-2748-4721-b5d0-ae6d607d85ba"). InnerVolumeSpecName "kube-api-access-q59rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:36:03.834636 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.834614    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c949ede-2748-4721-b5d0-ae6d607d85ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:36:03.834636 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.834638    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c949ede-2748-4721-b5d0-ae6d607d85ba-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:36:03.834775 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.834647    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c949ede-2748-4721-b5d0-ae6d607d85ba-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:36:03.834775 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:03.834655    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q59rs\" (UniqueName: \"kubernetes.io/projected/3c949ede-2748-4721-b5d0-ae6d607d85ba-kube-api-access-q59rs\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:36:04.012041 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.011962    2560 generic.go:356] "Generic (PLEG): container finished" podID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerID="fcdb0a55e803e53bba675fd078e7067d7c01db7f4aa5c84f11f4a74b7e68982a" exitCode=0
Jul 24 21:36:04.012194 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.012035    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" event={"ID":"0748c7ef-2143-4a24-8c58-44c360b6f9f1","Type":"ContainerDied","Data":"fcdb0a55e803e53bba675fd078e7067d7c01db7f4aa5c84f11f4a74b7e68982a"}
Jul 24 21:36:04.013968 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.013945    2560 generic.go:356] "Generic (PLEG): container finished" podID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerID="3ec1d13a2a1176bc2b6d0ae33db5b10863b84a6f14e0d7cac9553456f5a001bf" exitCode=0
Jul 24 21:36:04.014072 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.013972    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" event={"ID":"3c949ede-2748-4721-b5d0-ae6d607d85ba","Type":"ContainerDied","Data":"3ec1d13a2a1176bc2b6d0ae33db5b10863b84a6f14e0d7cac9553456f5a001bf"}
Jul 24 21:36:04.014072 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.013995    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"
Jul 24 21:36:04.014072 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.014020    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8" event={"ID":"3c949ede-2748-4721-b5d0-ae6d607d85ba","Type":"ContainerDied","Data":"087b151e75f4ba61a2755ca445004dc2d7d957e2b7542a0b80465b6b40fda0ca"}
Jul 24 21:36:04.014072 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.014049    2560 scope.go:117] "RemoveContainer" containerID="187e12cce7e5dc07a38d60f4cd067c91a697f22c12b2b9769126b45985fe3653"
Jul 24 21:36:04.015891 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.015865    2560 generic.go:356] "Generic (PLEG): container finished" podID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerID="ee3ac5867d031975c90481082111a4cea2d96a40037a46365eb610a6a77ab82b" exitCode=0
Jul 24 21:36:04.015996 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.015949    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" event={"ID":"a5660e06-3b68-4753-a86e-9b09d40dc645","Type":"ContainerDied","Data":"ee3ac5867d031975c90481082111a4cea2d96a40037a46365eb610a6a77ab82b"}
Jul 24 21:36:04.024835 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.024818    2560 scope.go:117] "RemoveContainer" containerID="3ec1d13a2a1176bc2b6d0ae33db5b10863b84a6f14e0d7cac9553456f5a001bf"
Jul 24 21:36:04.034301 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.034264    2560 scope.go:117] "RemoveContainer" containerID="ce2a4dc7268ca267c843d9a0737591423ddb679225f411057042e7ed8bc4a2ce"
Jul 24 21:36:04.052624 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.052604    2560 scope.go:117] "RemoveContainer" containerID="187e12cce7e5dc07a38d60f4cd067c91a697f22c12b2b9769126b45985fe3653"
Jul 24 21:36:04.052967 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:36:04.052946    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187e12cce7e5dc07a38d60f4cd067c91a697f22c12b2b9769126b45985fe3653\": container with ID starting with 187e12cce7e5dc07a38d60f4cd067c91a697f22c12b2b9769126b45985fe3653 not found: ID does not exist" containerID="187e12cce7e5dc07a38d60f4cd067c91a697f22c12b2b9769126b45985fe3653"
Jul 24 21:36:04.053049 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.052977    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187e12cce7e5dc07a38d60f4cd067c91a697f22c12b2b9769126b45985fe3653"} err="failed to get container status \"187e12cce7e5dc07a38d60f4cd067c91a697f22c12b2b9769126b45985fe3653\": rpc error: code = NotFound desc = could not find container \"187e12cce7e5dc07a38d60f4cd067c91a697f22c12b2b9769126b45985fe3653\": container with ID starting with 187e12cce7e5dc07a38d60f4cd067c91a697f22c12b2b9769126b45985fe3653 not found: ID does not exist"
Jul 24 21:36:04.053049 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.052996    2560 scope.go:117] "RemoveContainer" containerID="3ec1d13a2a1176bc2b6d0ae33db5b10863b84a6f14e0d7cac9553456f5a001bf"
Jul 24 21:36:04.053323 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:36:04.053302    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ec1d13a2a1176bc2b6d0ae33db5b10863b84a6f14e0d7cac9553456f5a001bf\": container with ID starting with 3ec1d13a2a1176bc2b6d0ae33db5b10863b84a6f14e0d7cac9553456f5a001bf not found: ID does not exist" containerID="3ec1d13a2a1176bc2b6d0ae33db5b10863b84a6f14e0d7cac9553456f5a001bf"
Jul 24 21:36:04.053409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.053332    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ec1d13a2a1176bc2b6d0ae33db5b10863b84a6f14e0d7cac9553456f5a001bf"} err="failed to get container status \"3ec1d13a2a1176bc2b6d0ae33db5b10863b84a6f14e0d7cac9553456f5a001bf\": rpc error: code = NotFound desc = could not find container \"3ec1d13a2a1176bc2b6d0ae33db5b10863b84a6f14e0d7cac9553456f5a001bf\": container with ID starting with 3ec1d13a2a1176bc2b6d0ae33db5b10863b84a6f14e0d7cac9553456f5a001bf not found: ID does not exist"
Jul 24 21:36:04.053409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.053352    2560 scope.go:117] "RemoveContainer" containerID="ce2a4dc7268ca267c843d9a0737591423ddb679225f411057042e7ed8bc4a2ce"
Jul 24 21:36:04.053663 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:36:04.053634    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2a4dc7268ca267c843d9a0737591423ddb679225f411057042e7ed8bc4a2ce\": container with ID starting with ce2a4dc7268ca267c843d9a0737591423ddb679225f411057042e7ed8bc4a2ce not found: ID does not exist" containerID="ce2a4dc7268ca267c843d9a0737591423ddb679225f411057042e7ed8bc4a2ce"
Jul 24 21:36:04.053752 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.053673    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2a4dc7268ca267c843d9a0737591423ddb679225f411057042e7ed8bc4a2ce"} err="failed to get container status \"ce2a4dc7268ca267c843d9a0737591423ddb679225f411057042e7ed8bc4a2ce\": rpc error: code = NotFound desc = could not find container \"ce2a4dc7268ca267c843d9a0737591423ddb679225f411057042e7ed8bc4a2ce\": container with ID starting with ce2a4dc7268ca267c843d9a0737591423ddb679225f411057042e7ed8bc4a2ce not found: ID does not exist"
Jul 24 21:36:04.062452 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.062431    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"]
Jul 24 21:36:04.065796 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:04.065773    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-6f3c9-predictor-85d796cbf-gkvg8"]
Jul 24 21:36:05.022780 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:05.022739    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" event={"ID":"0748c7ef-2143-4a24-8c58-44c360b6f9f1","Type":"ContainerStarted","Data":"44540d86d1b0d91c3b6e9e73c203a2329f31c7dcb8f17eb209c178442d6d7016"}
Jul 24 21:36:05.023276 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:05.022813    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" event={"ID":"0748c7ef-2143-4a24-8c58-44c360b6f9f1","Type":"ContainerStarted","Data":"7c42fb5ea55f0d2e45a6eb75cc8ef9dda56b8be0c25775f4e23607dfef3d485f"}
Jul 24 21:36:05.023276 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:05.023107    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:36:05.023385 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:05.023290    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:36:05.024582 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:05.024555    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 24 21:36:05.025986 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:05.025965    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" event={"ID":"a5660e06-3b68-4753-a86e-9b09d40dc645","Type":"ContainerStarted","Data":"ea8f53e4ffa6bc76577b617d3704de19dd73cbc8530352ae0432d5144f47938a"}
Jul 24 21:36:05.026091 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:05.026016    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" event={"ID":"a5660e06-3b68-4753-a86e-9b09d40dc645","Type":"ContainerStarted","Data":"879d374fcfb35afa0174149c4d387af5117a9e66139c974c53ec6e57f07a4116"}
Jul 24 21:36:05.026259 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:05.026240    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:36:05.043431 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:05.043384    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" podStartSLOduration=2.3150350729999998 podStartE2EDuration="6.04336552s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.728330447s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:36:05.042599091 +0000 UTC m=+1243.798944905" watchObservedRunningTime="2026-07-24 21:36:05.04336552 +0000 UTC m=+1243.799711335"
Jul 24 21:36:05.067696 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:05.067653    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" podStartSLOduration=2.5205506939999998 podStartE2EDuration="6.06763951s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.547088816s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:36:05.066401383 +0000 UTC m=+1243.822747196" watchObservedRunningTime="2026-07-24 21:36:05.06763951 +0000 UTC m=+1243.823985327"
Jul 24 21:36:05.753713 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:05.753677    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" path="/var/lib/kubelet/pods/3c949ede-2748-4721-b5d0-ae6d607d85ba/volumes"
Jul 24 21:36:06.030673 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:06.030582    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 24 21:36:06.031101 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:06.030831    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:36:06.032045 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:06.032020    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 24 21:36:07.034545 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:07.034507    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 24 21:36:11.035473 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:11.035448    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:36:11.036032 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:11.036006    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 24 21:36:12.040220 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:12.040188    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:36:12.040813 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:12.040785    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 24 21:36:21.036167 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:21.036099    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 24 21:36:22.041355 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:22.041254    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 24 21:36:31.036972 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:31.036929    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 24 21:36:32.041204 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:32.041163    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 24 21:36:41.036769 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:41.036733    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 24 21:36:42.041385 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:42.041342    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 24 21:36:51.035961 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:51.035923    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 24 21:36:52.041413 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:36:52.041376    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 24 21:37:01.036204 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:01.036163    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 24 21:37:02.040933 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:02.040895    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 24 21:37:11.036753 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:11.036707    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 24 21:37:12.040718 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:12.040679    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 24 21:37:14.749759 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:14.749715    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 24 21:37:21.753748 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:21.753722    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:37:24.750339 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:24.750303    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:37:29.243470 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:29.243442    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"]
Jul 24 21:37:29.243942 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:29.243792    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kserve-container" containerID="cri-o://879d374fcfb35afa0174149c4d387af5117a9e66139c974c53ec6e57f07a4116" gracePeriod=30
Jul 24 21:37:29.243942 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:29.243851    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kube-rbac-proxy" containerID="cri-o://ea8f53e4ffa6bc76577b617d3704de19dd73cbc8530352ae0432d5144f47938a" gracePeriod=30
Jul 24 21:37:29.259941 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:29.259915    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"]
Jul 24 21:37:29.260242 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:29.260204    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kserve-container" containerID="cri-o://7c42fb5ea55f0d2e45a6eb75cc8ef9dda56b8be0c25775f4e23607dfef3d485f" gracePeriod=30
Jul 24 21:37:29.260316 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:29.260287    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kube-rbac-proxy" containerID="cri-o://44540d86d1b0d91c3b6e9e73c203a2329f31c7dcb8f17eb209c178442d6d7016" gracePeriod=30
Jul 24 21:37:29.262790 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:29.262768    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:37:29.365031 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:29.364995    2560 generic.go:356] "Generic (PLEG): container finished" podID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerID="ea8f53e4ffa6bc76577b617d3704de19dd73cbc8530352ae0432d5144f47938a" exitCode=2
Jul 24 21:37:29.365204 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:29.365069    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" event={"ID":"a5660e06-3b68-4753-a86e-9b09d40dc645","Type":"ContainerDied","Data":"ea8f53e4ffa6bc76577b617d3704de19dd73cbc8530352ae0432d5144f47938a"}
Jul 24 21:37:30.371554 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:30.371522    2560 generic.go:356] "Generic (PLEG): container finished" podID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerID="44540d86d1b0d91c3b6e9e73c203a2329f31c7dcb8f17eb209c178442d6d7016" exitCode=2
Jul 24 21:37:30.371554 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:30.371567    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" event={"ID":"0748c7ef-2143-4a24-8c58-44c360b6f9f1","Type":"ContainerDied","Data":"44540d86d1b0d91c3b6e9e73c203a2329f31c7dcb8f17eb209c178442d6d7016"}
Jul 24 21:37:30.524607 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:30.524569    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:37:31.031440 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:31.031402    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 24 21:37:31.751653 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:31.751612    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 24 21:37:32.035080 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:32.034988    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 24 21:37:33.912922 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.912899    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:37:33.915554 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.915536    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:37:33.962602 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.962539    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5660e06-3b68-4753-a86e-9b09d40dc645-proxy-tls\") pod \"a5660e06-3b68-4753-a86e-9b09d40dc645\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") "
Jul 24 21:37:33.962602 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.962571    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5gc5\" (UniqueName: \"kubernetes.io/projected/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kube-api-access-k5gc5\") pod \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") "
Jul 24 21:37:33.962602 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.962595    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55w5b\" (UniqueName: \"kubernetes.io/projected/a5660e06-3b68-4753-a86e-9b09d40dc645-kube-api-access-55w5b\") pod \"a5660e06-3b68-4753-a86e-9b09d40dc645\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") "
Jul 24 21:37:33.962829 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.962618    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kserve-provision-location\") pod \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") "
Jul 24 21:37:33.962829 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.962742    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5660e06-3b68-4753-a86e-9b09d40dc645-kserve-provision-location\") pod \"a5660e06-3b68-4753-a86e-9b09d40dc645\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") "
Jul 24 21:37:33.962829 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.962792    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kube-rbac-proxy-sar-config\") pod \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") "
Jul 24 21:37:33.962984 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.962827    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0748c7ef-2143-4a24-8c58-44c360b6f9f1-proxy-tls\") pod \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") "
Jul 24 21:37:33.962984 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.962856    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5660e06-3b68-4753-a86e-9b09d40dc645-kube-rbac-proxy-sar-config\") pod \"a5660e06-3b68-4753-a86e-9b09d40dc645\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") "
Jul 24 21:37:33.962984 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.962888    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0748c7ef-2143-4a24-8c58-44c360b6f9f1-cabundle-cert\") pod \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\" (UID: \"0748c7ef-2143-4a24-8c58-44c360b6f9f1\") "
Jul 24 21:37:33.962984 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.962906    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0748c7ef-2143-4a24-8c58-44c360b6f9f1" (UID: "0748c7ef-2143-4a24-8c58-44c360b6f9f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:37:33.963822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.963250    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0748c7ef-2143-4a24-8c58-44c360b6f9f1" (UID: "0748c7ef-2143-4a24-8c58-44c360b6f9f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:37:33.963822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.963412    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0748c7ef-2143-4a24-8c58-44c360b6f9f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0748c7ef-2143-4a24-8c58-44c360b6f9f1" (UID: "0748c7ef-2143-4a24-8c58-44c360b6f9f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:37:33.963822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.963424    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5660e06-3b68-4753-a86e-9b09d40dc645-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a5660e06-3b68-4753-a86e-9b09d40dc645" (UID: "a5660e06-3b68-4753-a86e-9b09d40dc645"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:37:33.963822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.963502    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5660e06-3b68-4753-a86e-9b09d40dc645-cabundle-cert\") pod \"a5660e06-3b68-4753-a86e-9b09d40dc645\" (UID: \"a5660e06-3b68-4753-a86e-9b09d40dc645\") "
Jul 24 21:37:33.963822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.963783    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5660e06-3b68-4753-a86e-9b09d40dc645-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5660e06-3b68-4753-a86e-9b09d40dc645" (UID: "a5660e06-3b68-4753-a86e-9b09d40dc645"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:37:33.963822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.963807    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5660e06-3b68-4753-a86e-9b09d40dc645-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5660e06-3b68-4753-a86e-9b09d40dc645" (UID: "a5660e06-3b68-4753-a86e-9b09d40dc645"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:37:33.964475 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.964450    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:37:33.964554 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.964481    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5660e06-3b68-4753-a86e-9b09d40dc645-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:37:33.964554 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.964496    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0748c7ef-2143-4a24-8c58-44c360b6f9f1-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:37:33.964554 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.964509    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5660e06-3b68-4753-a86e-9b09d40dc645-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:37:33.964554 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.964522    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:37:33.964554 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.964535    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5660e06-3b68-4753-a86e-9b09d40dc645-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:37:33.965101 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.965079    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5660e06-3b68-4753-a86e-9b09d40dc645-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a5660e06-3b68-4753-a86e-9b09d40dc645" (UID: "a5660e06-3b68-4753-a86e-9b09d40dc645"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:37:33.965205 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.965186    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kube-api-access-k5gc5" (OuterVolumeSpecName: "kube-api-access-k5gc5") pod "0748c7ef-2143-4a24-8c58-44c360b6f9f1" (UID: "0748c7ef-2143-4a24-8c58-44c360b6f9f1"). InnerVolumeSpecName "kube-api-access-k5gc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:37:33.965562 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.965540    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0748c7ef-2143-4a24-8c58-44c360b6f9f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0748c7ef-2143-4a24-8c58-44c360b6f9f1" (UID: "0748c7ef-2143-4a24-8c58-44c360b6f9f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:37:33.965562 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:33.965550    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5660e06-3b68-4753-a86e-9b09d40dc645-kube-api-access-55w5b" (OuterVolumeSpecName: "kube-api-access-55w5b") pod "a5660e06-3b68-4753-a86e-9b09d40dc645" (UID: "a5660e06-3b68-4753-a86e-9b09d40dc645"). InnerVolumeSpecName "kube-api-access-55w5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:37:34.065088 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.065061    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5660e06-3b68-4753-a86e-9b09d40dc645-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:37:34.065088 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.065085    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5gc5\" (UniqueName: \"kubernetes.io/projected/0748c7ef-2143-4a24-8c58-44c360b6f9f1-kube-api-access-k5gc5\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:37:34.065305 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.065096    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55w5b\" (UniqueName: \"kubernetes.io/projected/a5660e06-3b68-4753-a86e-9b09d40dc645-kube-api-access-55w5b\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:37:34.065305 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.065108    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0748c7ef-2143-4a24-8c58-44c360b6f9f1-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:37:34.304344 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.304223    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:37:34.373371 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.373299    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:37:34.376989 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.376967    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"]
Jul 24 21:37:34.377278 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377266    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kserve-container"
Jul 24 21:37:34.377325 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377280    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kserve-container"
Jul 24 21:37:34.377325 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377289    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kube-rbac-proxy"
Jul 24 21:37:34.377325 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377295    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kube-rbac-proxy"
Jul 24 21:37:34.377325 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377303    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="storage-initializer"
Jul 24 21:37:34.377325 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377309    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="storage-initializer"
Jul 24 21:37:34.377325 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377316    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container"
Jul 24 21:37:34.377325 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377321    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377328    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="storage-initializer"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377334    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="storage-initializer"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377343    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kube-rbac-proxy"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377348    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kube-rbac-proxy"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377355    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="storage-initializer"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377360    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="storage-initializer"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377368    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kserve-container"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377373    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kserve-container"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377384    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kube-rbac-proxy"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377388    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kube-rbac-proxy"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377435    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kube-rbac-proxy"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377442    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c949ede-2748-4721-b5d0-ae6d607d85ba" containerName="kserve-container"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377447    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kserve-container"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377454    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kube-rbac-proxy"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377458    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerName="kube-rbac-proxy"
Jul 24 21:37:34.377556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.377464    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerName="kserve-container"
Jul 24 21:37:34.381711 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.381694    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"]
Jul 24 21:37:34.381830 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.381813    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:34.384533 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.384468    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 24 21:37:34.385257 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.384761    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 24 21:37:34.386760 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.386365    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.388856 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.388459    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 24 21:37:34.393276 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.393253    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"]
Jul 24 21:37:34.393941 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.393914    2560 generic.go:356] "Generic (PLEG): container finished" podID="a5660e06-3b68-4753-a86e-9b09d40dc645" containerID="879d374fcfb35afa0174149c4d387af5117a9e66139c974c53ec6e57f07a4116" exitCode=0
Jul 24 21:37:34.394050 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.394003    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" event={"ID":"a5660e06-3b68-4753-a86e-9b09d40dc645","Type":"ContainerDied","Data":"879d374fcfb35afa0174149c4d387af5117a9e66139c974c53ec6e57f07a4116"}
Jul 24 21:37:34.394141 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.394016    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"
Jul 24 21:37:34.394141 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.394063    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2" event={"ID":"a5660e06-3b68-4753-a86e-9b09d40dc645","Type":"ContainerDied","Data":"1836e90c02703e25f9f93b3732d4525bc187852bf65a78b19504ec599f8ddd75"}
Jul 24 21:37:34.394141 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.394099    2560 scope.go:117] "RemoveContainer" containerID="ea8f53e4ffa6bc76577b617d3704de19dd73cbc8530352ae0432d5144f47938a"
Jul 24 21:37:34.398822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.398794    2560 generic.go:356] "Generic (PLEG): container finished" podID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" containerID="7c42fb5ea55f0d2e45a6eb75cc8ef9dda56b8be0c25775f4e23607dfef3d485f" exitCode=0
Jul 24 21:37:34.398945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.398853    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" event={"ID":"0748c7ef-2143-4a24-8c58-44c360b6f9f1","Type":"ContainerDied","Data":"7c42fb5ea55f0d2e45a6eb75cc8ef9dda56b8be0c25775f4e23607dfef3d485f"}
Jul 24 21:37:34.398945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.398902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v" event={"ID":"0748c7ef-2143-4a24-8c58-44c360b6f9f1","Type":"ContainerDied","Data":"77b1ab11bf260fe498396b8b294a22e24b940bb788a39dd41d7a2faf8a6395a5"}
Jul 24 21:37:34.399104 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.399037    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"
Jul 24 21:37:34.409146 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.409109    2560 scope.go:117] "RemoveContainer" containerID="879d374fcfb35afa0174149c4d387af5117a9e66139c974c53ec6e57f07a4116"
Jul 24 21:37:34.410777 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.410570    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"]
Jul 24 21:37:34.421182 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.421163    2560 scope.go:117] "RemoveContainer" containerID="ee3ac5867d031975c90481082111a4cea2d96a40037a46365eb610a6a77ab82b"
Jul 24 21:37:34.429014 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.429000    2560 scope.go:117] "RemoveContainer" containerID="ea8f53e4ffa6bc76577b617d3704de19dd73cbc8530352ae0432d5144f47938a"
Jul 24 21:37:34.429325 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:37:34.429309    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8f53e4ffa6bc76577b617d3704de19dd73cbc8530352ae0432d5144f47938a\": container with ID starting with ea8f53e4ffa6bc76577b617d3704de19dd73cbc8530352ae0432d5144f47938a not found: ID does not exist" containerID="ea8f53e4ffa6bc76577b617d3704de19dd73cbc8530352ae0432d5144f47938a"
Jul 24 21:37:34.429400 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.429334    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8f53e4ffa6bc76577b617d3704de19dd73cbc8530352ae0432d5144f47938a"} err="failed to get container status \"ea8f53e4ffa6bc76577b617d3704de19dd73cbc8530352ae0432d5144f47938a\": rpc error: code = NotFound desc = could not find container \"ea8f53e4ffa6bc76577b617d3704de19dd73cbc8530352ae0432d5144f47938a\": container with ID starting with ea8f53e4ffa6bc76577b617d3704de19dd73cbc8530352ae0432d5144f47938a not found: ID does not exist"
Jul 24 21:37:34.429400 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.429351    2560 scope.go:117] "RemoveContainer" containerID="879d374fcfb35afa0174149c4d387af5117a9e66139c974c53ec6e57f07a4116"
Jul 24 21:37:34.429576 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:37:34.429557    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"879d374fcfb35afa0174149c4d387af5117a9e66139c974c53ec6e57f07a4116\": container with ID starting with 879d374fcfb35afa0174149c4d387af5117a9e66139c974c53ec6e57f07a4116 not found: ID does not exist" containerID="879d374fcfb35afa0174149c4d387af5117a9e66139c974c53ec6e57f07a4116"
Jul 24 21:37:34.429641 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.429585    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"879d374fcfb35afa0174149c4d387af5117a9e66139c974c53ec6e57f07a4116"} err="failed to get container status \"879d374fcfb35afa0174149c4d387af5117a9e66139c974c53ec6e57f07a4116\": rpc error: code = NotFound desc = could not find container \"879d374fcfb35afa0174149c4d387af5117a9e66139c974c53ec6e57f07a4116\": container with ID starting with 879d374fcfb35afa0174149c4d387af5117a9e66139c974c53ec6e57f07a4116 not found: ID does not exist"
Jul 24 21:37:34.429641 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.429608    2560 scope.go:117] "RemoveContainer" containerID="ee3ac5867d031975c90481082111a4cea2d96a40037a46365eb610a6a77ab82b"
Jul 24 21:37:34.429945 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:37:34.429921    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee3ac5867d031975c90481082111a4cea2d96a40037a46365eb610a6a77ab82b\": container with ID starting with ee3ac5867d031975c90481082111a4cea2d96a40037a46365eb610a6a77ab82b not found: ID does not exist" containerID="ee3ac5867d031975c90481082111a4cea2d96a40037a46365eb610a6a77ab82b"
Jul 24 21:37:34.430041 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.429953    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee3ac5867d031975c90481082111a4cea2d96a40037a46365eb610a6a77ab82b"} err="failed to get container status \"ee3ac5867d031975c90481082111a4cea2d96a40037a46365eb610a6a77ab82b\": rpc error: code = NotFound desc = could not find container \"ee3ac5867d031975c90481082111a4cea2d96a40037a46365eb610a6a77ab82b\": container with ID starting with ee3ac5867d031975c90481082111a4cea2d96a40037a46365eb610a6a77ab82b not found: ID does not exist"
Jul 24 21:37:34.430041 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.429976    2560 scope.go:117] "RemoveContainer" containerID="44540d86d1b0d91c3b6e9e73c203a2329f31c7dcb8f17eb209c178442d6d7016"
Jul 24 21:37:34.438099 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.438084    2560 scope.go:117] "RemoveContainer" containerID="7c42fb5ea55f0d2e45a6eb75cc8ef9dda56b8be0c25775f4e23607dfef3d485f"
Jul 24 21:37:34.445699 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.445677    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"]
Jul 24 21:37:34.446521 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.446505    2560 scope.go:117] "RemoveContainer" containerID="fcdb0a55e803e53bba675fd078e7067d7c01db7f4aa5c84f11f4a74b7e68982a"
Jul 24 21:37:34.447568 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.447552    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7f74cfd478-4vq6v"]
Jul 24 21:37:34.453829 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.453814    2560 scope.go:117] "RemoveContainer" containerID="44540d86d1b0d91c3b6e9e73c203a2329f31c7dcb8f17eb209c178442d6d7016"
Jul 24 21:37:34.454070 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:37:34.454050    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44540d86d1b0d91c3b6e9e73c203a2329f31c7dcb8f17eb209c178442d6d7016\": container with ID starting with 44540d86d1b0d91c3b6e9e73c203a2329f31c7dcb8f17eb209c178442d6d7016 not found: ID does not exist" containerID="44540d86d1b0d91c3b6e9e73c203a2329f31c7dcb8f17eb209c178442d6d7016"
Jul 24 21:37:34.454230 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.454077    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44540d86d1b0d91c3b6e9e73c203a2329f31c7dcb8f17eb209c178442d6d7016"} err="failed to get container status \"44540d86d1b0d91c3b6e9e73c203a2329f31c7dcb8f17eb209c178442d6d7016\": rpc error: code = NotFound desc = could not find container \"44540d86d1b0d91c3b6e9e73c203a2329f31c7dcb8f17eb209c178442d6d7016\": container with ID starting with 44540d86d1b0d91c3b6e9e73c203a2329f31c7dcb8f17eb209c178442d6d7016 not found: ID does not exist"
Jul 24 21:37:34.454230 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.454100    2560 scope.go:117] "RemoveContainer" containerID="7c42fb5ea55f0d2e45a6eb75cc8ef9dda56b8be0c25775f4e23607dfef3d485f"
Jul 24 21:37:34.454427 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:37:34.454411    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c42fb5ea55f0d2e45a6eb75cc8ef9dda56b8be0c25775f4e23607dfef3d485f\": container with ID starting with 7c42fb5ea55f0d2e45a6eb75cc8ef9dda56b8be0c25775f4e23607dfef3d485f not found: ID does not exist" containerID="7c42fb5ea55f0d2e45a6eb75cc8ef9dda56b8be0c25775f4e23607dfef3d485f"
Jul 24 21:37:34.454469 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.454433    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c42fb5ea55f0d2e45a6eb75cc8ef9dda56b8be0c25775f4e23607dfef3d485f"} err="failed to get container status \"7c42fb5ea55f0d2e45a6eb75cc8ef9dda56b8be0c25775f4e23607dfef3d485f\": rpc error: code = NotFound desc = could not find container \"7c42fb5ea55f0d2e45a6eb75cc8ef9dda56b8be0c25775f4e23607dfef3d485f\": container with ID starting with 7c42fb5ea55f0d2e45a6eb75cc8ef9dda56b8be0c25775f4e23607dfef3d485f not found: ID does not exist"
Jul 24 21:37:34.454469 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.454458    2560 scope.go:117] "RemoveContainer" containerID="fcdb0a55e803e53bba675fd078e7067d7c01db7f4aa5c84f11f4a74b7e68982a"
Jul 24 21:37:34.454715 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:37:34.454698    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcdb0a55e803e53bba675fd078e7067d7c01db7f4aa5c84f11f4a74b7e68982a\": container with ID starting with fcdb0a55e803e53bba675fd078e7067d7c01db7f4aa5c84f11f4a74b7e68982a not found: ID does not exist" containerID="fcdb0a55e803e53bba675fd078e7067d7c01db7f4aa5c84f11f4a74b7e68982a"
Jul 24 21:37:34.454761 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.454720    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcdb0a55e803e53bba675fd078e7067d7c01db7f4aa5c84f11f4a74b7e68982a"} err="failed to get container status \"fcdb0a55e803e53bba675fd078e7067d7c01db7f4aa5c84f11f4a74b7e68982a\": rpc error: code = NotFound desc = could not find container \"fcdb0a55e803e53bba675fd078e7067d7c01db7f4aa5c84f11f4a74b7e68982a\": container with ID starting with fcdb0a55e803e53bba675fd078e7067d7c01db7f4aa5c84f11f4a74b7e68982a not found: ID does not exist"
Jul 24 21:37:34.459421 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.459400    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"]
Jul 24 21:37:34.463397 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.463368    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-75c7994655-5v8h2"]
Jul 24 21:37:34.468098 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.468078    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/609cfeac-e91b-4503-b8fa-d65d560a6281-kserve-provision-location\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:34.468232 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.468113    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/609cfeac-e91b-4503-b8fa-d65d560a6281-cabundle-cert\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:34.468232 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.468197    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnsgr\" (UniqueName: \"kubernetes.io/projected/609cfeac-e91b-4503-b8fa-d65d560a6281-kube-api-access-dnsgr\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:34.468232 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.468227    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77fd1550-ef43-41cc-8a2d-356ea250de44-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.468409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.468252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hlnt\" (UniqueName: \"kubernetes.io/projected/77fd1550-ef43-41cc-8a2d-356ea250de44-kube-api-access-8hlnt\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.468409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.468294    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77fd1550-ef43-41cc-8a2d-356ea250de44-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.468409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.468329    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77fd1550-ef43-41cc-8a2d-356ea250de44-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.468409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.468368    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77fd1550-ef43-41cc-8a2d-356ea250de44-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.468562 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.468411    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/609cfeac-e91b-4503-b8fa-d65d560a6281-proxy-tls\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:34.468562 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.468481    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/609cfeac-e91b-4503-b8fa-d65d560a6281-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:34.560565 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.560447    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:37:34.569406 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.569378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77fd1550-ef43-41cc-8a2d-356ea250de44-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.569568 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.569420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/609cfeac-e91b-4503-b8fa-d65d560a6281-proxy-tls\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:34.569568 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.569467    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/609cfeac-e91b-4503-b8fa-d65d560a6281-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:34.569568 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.569501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/609cfeac-e91b-4503-b8fa-d65d560a6281-kserve-provision-location\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:34.569568 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:37:34.569547    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 24 21:37:34.569818 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.569625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/609cfeac-e91b-4503-b8fa-d65d560a6281-cabundle-cert\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:34.569818 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:37:34.569704    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/609cfeac-e91b-4503-b8fa-d65d560a6281-proxy-tls podName:609cfeac-e91b-4503-b8fa-d65d560a6281 nodeName:}" failed. No retries permitted until 2026-07-24 21:37:35.069682517 +0000 UTC m=+1333.826028308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/609cfeac-e91b-4503-b8fa-d65d560a6281-proxy-tls") pod "isvc-canary-promote-predictor-74874cc84b-hl285" (UID: "609cfeac-e91b-4503-b8fa-d65d560a6281") : secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 24 21:37:34.570023 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.569867    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnsgr\" (UniqueName: \"kubernetes.io/projected/609cfeac-e91b-4503-b8fa-d65d560a6281-kube-api-access-dnsgr\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:34.570023 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.569877    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/609cfeac-e91b-4503-b8fa-d65d560a6281-kserve-provision-location\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:34.570023 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.569903    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77fd1550-ef43-41cc-8a2d-356ea250de44-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.570023 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.569931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hlnt\" (UniqueName: \"kubernetes.io/projected/77fd1550-ef43-41cc-8a2d-356ea250de44-kube-api-access-8hlnt\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.570023 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.569978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77fd1550-ef43-41cc-8a2d-356ea250de44-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.570023 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.570013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77fd1550-ef43-41cc-8a2d-356ea250de44-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.570463 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.570262    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/609cfeac-e91b-4503-b8fa-d65d560a6281-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:34.570463 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.570360    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/609cfeac-e91b-4503-b8fa-d65d560a6281-cabundle-cert\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:34.570463 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.570374    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77fd1550-ef43-41cc-8a2d-356ea250de44-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.570611 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.570528    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77fd1550-ef43-41cc-8a2d-356ea250de44-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.570611 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.570583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77fd1550-ef43-41cc-8a2d-356ea250de44-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.572659 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.572641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77fd1550-ef43-41cc-8a2d-356ea250de44-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.580412 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.580392    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnsgr\" (UniqueName: \"kubernetes.io/projected/609cfeac-e91b-4503-b8fa-d65d560a6281-kube-api-access-dnsgr\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:34.580504 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.580394    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hlnt\" (UniqueName: \"kubernetes.io/projected/77fd1550-ef43-41cc-8a2d-356ea250de44-kube-api-access-8hlnt\") pod \"isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.659682 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.659650    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:37:34.717822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.717796    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:34.860037 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:34.860008    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"]
Jul 24 21:37:34.863446 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:37:34.863417    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fd1550_ef43_41cc_8a2d_356ea250de44.slice/crio-bde07b0a546b582ab2780482ee23cbeea70ab979d4f4d1c21c956621004576dc WatchSource:0}: Error finding container bde07b0a546b582ab2780482ee23cbeea70ab979d4f4d1c21c956621004576dc: Status 404 returned error can't find the container with id bde07b0a546b582ab2780482ee23cbeea70ab979d4f4d1c21c956621004576dc
Jul 24 21:37:35.073979 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:35.073944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/609cfeac-e91b-4503-b8fa-d65d560a6281-proxy-tls\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:35.076657 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:35.076634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/609cfeac-e91b-4503-b8fa-d65d560a6281-proxy-tls\") pod \"isvc-canary-promote-predictor-74874cc84b-hl285\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:35.303900 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:35.303825    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:35.407147 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:35.407094    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" event={"ID":"77fd1550-ef43-41cc-8a2d-356ea250de44","Type":"ContainerStarted","Data":"eacc7ba6197bf166268436444f085ec9dff52783fa830a85efdac5d2376c8c51"}
Jul 24 21:37:35.407301 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:35.407191    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" event={"ID":"77fd1550-ef43-41cc-8a2d-356ea250de44","Type":"ContainerStarted","Data":"bde07b0a546b582ab2780482ee23cbeea70ab979d4f4d1c21c956621004576dc"}
Jul 24 21:37:35.466736 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:35.466703    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"]
Jul 24 21:37:35.469701 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:37:35.469677    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609cfeac_e91b_4503_b8fa_d65d560a6281.slice/crio-baea21064cf6fd030c3b6ddb5a15d3515673face01e64cd580a0ea2690a25e72 WatchSource:0}: Error finding container baea21064cf6fd030c3b6ddb5a15d3515673face01e64cd580a0ea2690a25e72: Status 404 returned error can't find the container with id baea21064cf6fd030c3b6ddb5a15d3515673face01e64cd580a0ea2690a25e72
Jul 24 21:37:35.753822 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:35.753790    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0748c7ef-2143-4a24-8c58-44c360b6f9f1" path="/var/lib/kubelet/pods/0748c7ef-2143-4a24-8c58-44c360b6f9f1/volumes"
Jul 24 21:37:35.754290 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:35.754276    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5660e06-3b68-4753-a86e-9b09d40dc645" path="/var/lib/kubelet/pods/a5660e06-3b68-4753-a86e-9b09d40dc645/volumes"
Jul 24 21:37:36.413644 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:36.413611    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" event={"ID":"609cfeac-e91b-4503-b8fa-d65d560a6281","Type":"ContainerStarted","Data":"15148889624aa3313a3305c0a486e3721952c99377ffe42dbabe533912dc1d3b"}
Jul 24 21:37:36.414003 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:36.413673    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" event={"ID":"609cfeac-e91b-4503-b8fa-d65d560a6281","Type":"ContainerStarted","Data":"baea21064cf6fd030c3b6ddb5a15d3515673face01e64cd580a0ea2690a25e72"}
Jul 24 21:37:39.430104 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:39.430020    2560 generic.go:356] "Generic (PLEG): container finished" podID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerID="eacc7ba6197bf166268436444f085ec9dff52783fa830a85efdac5d2376c8c51" exitCode=0
Jul 24 21:37:39.430104 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:39.430093    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" event={"ID":"77fd1550-ef43-41cc-8a2d-356ea250de44","Type":"ContainerDied","Data":"eacc7ba6197bf166268436444f085ec9dff52783fa830a85efdac5d2376c8c51"}
Jul 24 21:37:39.431759 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:39.431696    2560 generic.go:356] "Generic (PLEG): container finished" podID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerID="15148889624aa3313a3305c0a486e3721952c99377ffe42dbabe533912dc1d3b" exitCode=0
Jul 24 21:37:39.431814 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:39.431764    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" event={"ID":"609cfeac-e91b-4503-b8fa-d65d560a6281","Type":"ContainerDied","Data":"15148889624aa3313a3305c0a486e3721952c99377ffe42dbabe533912dc1d3b"}
Jul 24 21:37:40.438958 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:40.438920    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" event={"ID":"77fd1550-ef43-41cc-8a2d-356ea250de44","Type":"ContainerStarted","Data":"aa1f7924c934efa8f997f92592760f82da251f6271232d3d57e680a57f8ec95c"}
Jul 24 21:37:40.439418 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:40.438987    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" event={"ID":"77fd1550-ef43-41cc-8a2d-356ea250de44","Type":"ContainerStarted","Data":"f275fdd96de1caf63904ccf03fd0954932d22fa7b09a7d0de1682e1f945b2d47"}
Jul 24 21:37:40.439418 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:40.439272    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:40.439527 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:40.439437    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:40.440780 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:40.440751    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 24 21:37:40.441174 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:40.441155    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" event={"ID":"609cfeac-e91b-4503-b8fa-d65d560a6281","Type":"ContainerStarted","Data":"4573ac8df49930ba0336d14e544c61c94c3679d6f26a701420668596cd22aa2b"}
Jul 24 21:37:40.441232 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:40.441202    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" event={"ID":"609cfeac-e91b-4503-b8fa-d65d560a6281","Type":"ContainerStarted","Data":"5b8fab641939914486c983200c4040eb7846a9381a688468ebe912701ffff464"}
Jul 24 21:37:40.441451 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:40.441432    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:40.441557 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:40.441475    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:40.442351 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:40.442330    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 24 21:37:40.462450 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:40.462412    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" podStartSLOduration=2.468739745 podStartE2EDuration="6.462397719s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.993657974s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:37:40.461718502 +0000 UTC m=+1339.218064312" watchObservedRunningTime="2026-07-24 21:37:40.462397719 +0000 UTC m=+1339.218743532"
Jul 24 21:37:40.484255 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:40.484206    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" podStartSLOduration=2.867898155 podStartE2EDuration="6.484193075s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.61629492s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:37:40.483719486 +0000 UTC m=+1339.240065300" watchObservedRunningTime="2026-07-24 21:37:40.484193075 +0000 UTC m=+1339.240538888"
Jul 24 21:37:41.451551 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:41.451513    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 24 21:37:41.451946 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:41.451661    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 24 21:37:46.458448 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:46.458407    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:37:46.458861 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:46.458503    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:37:46.459004 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:46.458978    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 24 21:37:46.459155 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:46.459061    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 24 21:37:56.459928 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:56.459887    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 24 21:37:56.460327 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:37:56.459887    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 24 21:38:06.458914 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:38:06.458876    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 24 21:38:06.459373 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:38:06.458890    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 24 21:38:16.459328 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:38:16.459293    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 24 21:38:16.459826 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:38:16.459290    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 24 21:38:26.459729 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:38:26.459684    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 24 21:38:26.460153 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:38:26.459685    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 24 21:38:36.458921 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:38:36.458888    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 24 21:38:36.459326 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:38:36.458883    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 24 21:38:46.458880 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:38:46.458834    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 24 21:38:46.459370 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:38:46.458973    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 24 21:38:56.459905 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:38:56.459869    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:38:56.460330 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:38:56.459953    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:39:04.487914 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:04.487876    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"]
Jul 24 21:39:04.488428 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:04.488296    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kserve-container" containerID="cri-o://5b8fab641939914486c983200c4040eb7846a9381a688468ebe912701ffff464" gracePeriod=30
Jul 24 21:39:04.488428 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:04.488342    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kube-rbac-proxy" containerID="cri-o://4573ac8df49930ba0336d14e544c61c94c3679d6f26a701420668596cd22aa2b" gracePeriod=30
Jul 24 21:39:04.509067 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:04.509035    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:39:04.789448 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:04.789366    2560 generic.go:356] "Generic (PLEG): container finished" podID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerID="4573ac8df49930ba0336d14e544c61c94c3679d6f26a701420668596cd22aa2b" exitCode=2
Jul 24 21:39:04.789448 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:04.789402    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" event={"ID":"609cfeac-e91b-4503-b8fa-d65d560a6281","Type":"ContainerDied","Data":"4573ac8df49930ba0336d14e544c61c94c3679d6f26a701420668596cd22aa2b"}
Jul 24 21:39:06.452313 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:06.452264    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 24 21:39:06.459547 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:06.459506    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 24 21:39:08.807762 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:08.807733    2560 generic.go:356] "Generic (PLEG): container finished" podID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerID="5b8fab641939914486c983200c4040eb7846a9381a688468ebe912701ffff464" exitCode=0
Jul 24 21:39:08.808026 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:08.807797    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" event={"ID":"609cfeac-e91b-4503-b8fa-d65d560a6281","Type":"ContainerDied","Data":"5b8fab641939914486c983200c4040eb7846a9381a688468ebe912701ffff464"}
Jul 24 21:39:08.934182 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:08.934157    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:39:08.984724 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:08.984697    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/609cfeac-e91b-4503-b8fa-d65d560a6281-cabundle-cert\") pod \"609cfeac-e91b-4503-b8fa-d65d560a6281\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") "
Jul 24 21:39:08.984886 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:08.984748    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnsgr\" (UniqueName: \"kubernetes.io/projected/609cfeac-e91b-4503-b8fa-d65d560a6281-kube-api-access-dnsgr\") pod \"609cfeac-e91b-4503-b8fa-d65d560a6281\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") "
Jul 24 21:39:08.984886 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:08.984776    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/609cfeac-e91b-4503-b8fa-d65d560a6281-proxy-tls\") pod \"609cfeac-e91b-4503-b8fa-d65d560a6281\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") "
Jul 24 21:39:08.984886 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:08.984801    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/609cfeac-e91b-4503-b8fa-d65d560a6281-kube-rbac-proxy-sar-config\") pod \"609cfeac-e91b-4503-b8fa-d65d560a6281\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") "
Jul 24 21:39:08.984886 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:08.984834    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/609cfeac-e91b-4503-b8fa-d65d560a6281-kserve-provision-location\") pod \"609cfeac-e91b-4503-b8fa-d65d560a6281\" (UID: \"609cfeac-e91b-4503-b8fa-d65d560a6281\") "
Jul 24 21:39:08.985266 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:08.985208    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609cfeac-e91b-4503-b8fa-d65d560a6281-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "609cfeac-e91b-4503-b8fa-d65d560a6281" (UID: "609cfeac-e91b-4503-b8fa-d65d560a6281"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:39:08.985266 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:08.985216    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609cfeac-e91b-4503-b8fa-d65d560a6281-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "609cfeac-e91b-4503-b8fa-d65d560a6281" (UID: "609cfeac-e91b-4503-b8fa-d65d560a6281"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:39:08.985266 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:08.985216    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609cfeac-e91b-4503-b8fa-d65d560a6281-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "609cfeac-e91b-4503-b8fa-d65d560a6281" (UID: "609cfeac-e91b-4503-b8fa-d65d560a6281"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:39:08.986974 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:08.986948    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/609cfeac-e91b-4503-b8fa-d65d560a6281-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "609cfeac-e91b-4503-b8fa-d65d560a6281" (UID: "609cfeac-e91b-4503-b8fa-d65d560a6281"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:39:08.987092 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:08.986977    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609cfeac-e91b-4503-b8fa-d65d560a6281-kube-api-access-dnsgr" (OuterVolumeSpecName: "kube-api-access-dnsgr") pod "609cfeac-e91b-4503-b8fa-d65d560a6281" (UID: "609cfeac-e91b-4503-b8fa-d65d560a6281"). InnerVolumeSpecName "kube-api-access-dnsgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:39:09.086006 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:09.085932    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/609cfeac-e91b-4503-b8fa-d65d560a6281-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:39:09.086006 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:09.085959    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnsgr\" (UniqueName: \"kubernetes.io/projected/609cfeac-e91b-4503-b8fa-d65d560a6281-kube-api-access-dnsgr\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:39:09.086006 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:09.085969    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/609cfeac-e91b-4503-b8fa-d65d560a6281-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:39:09.086006 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:09.085980    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/609cfeac-e91b-4503-b8fa-d65d560a6281-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:39:09.086006 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:09.085988    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/609cfeac-e91b-4503-b8fa-d65d560a6281-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:39:09.813258 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:09.813223    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285" event={"ID":"609cfeac-e91b-4503-b8fa-d65d560a6281","Type":"ContainerDied","Data":"baea21064cf6fd030c3b6ddb5a15d3515673face01e64cd580a0ea2690a25e72"}
Jul 24 21:39:09.813702 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:09.813285    2560 scope.go:117] "RemoveContainer" containerID="4573ac8df49930ba0336d14e544c61c94c3679d6f26a701420668596cd22aa2b"
Jul 24 21:39:09.813702 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:09.813236    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"
Jul 24 21:39:09.822934 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:09.822901    2560 scope.go:117] "RemoveContainer" containerID="5b8fab641939914486c983200c4040eb7846a9381a688468ebe912701ffff464"
Jul 24 21:39:09.830292 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:09.830275    2560 scope.go:117] "RemoveContainer" containerID="15148889624aa3313a3305c0a486e3721952c99377ffe42dbabe533912dc1d3b"
Jul 24 21:39:09.853206 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:09.853176    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"]
Jul 24 21:39:09.863990 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:09.863968    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-74874cc84b-hl285"]
Jul 24 21:39:11.759015 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:39:11.758984    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" path="/var/lib/kubelet/pods/609cfeac-e91b-4503-b8fa-d65d560a6281/volumes"
Jul 24 21:40:21.722986 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:21.722960    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:40:21.724622 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:21.724601    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:40:44.750147 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:44.750094    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"]
Jul 24 21:40:44.752792 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:44.750342    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kserve-container" containerID="cri-o://f275fdd96de1caf63904ccf03fd0954932d22fa7b09a7d0de1682e1f945b2d47" gracePeriod=30
Jul 24 21:40:44.752792 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:44.750615    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kube-rbac-proxy" containerID="cri-o://aa1f7924c934efa8f997f92592760f82da251f6271232d3d57e680a57f8ec95c" gracePeriod=30
Jul 24 21:40:45.178003 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:45.177967    2560 generic.go:356] "Generic (PLEG): container finished" podID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerID="aa1f7924c934efa8f997f92592760f82da251f6271232d3d57e680a57f8ec95c" exitCode=2
Jul 24 21:40:45.178197 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:45.178038    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" event={"ID":"77fd1550-ef43-41cc-8a2d-356ea250de44","Type":"ContainerDied","Data":"aa1f7924c934efa8f997f92592760f82da251f6271232d3d57e680a57f8ec95c"}
Jul 24 21:40:45.514679 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:45.514597    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:40:46.452434 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:46.452394    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 24 21:40:46.459556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:46.459531    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 24 21:40:49.196624 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.196596    2560 generic.go:356] "Generic (PLEG): container finished" podID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerID="f275fdd96de1caf63904ccf03fd0954932d22fa7b09a7d0de1682e1f945b2d47" exitCode=0
Jul 24 21:40:49.196946 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.196663    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" event={"ID":"77fd1550-ef43-41cc-8a2d-356ea250de44","Type":"ContainerDied","Data":"f275fdd96de1caf63904ccf03fd0954932d22fa7b09a7d0de1682e1f945b2d47"}
Jul 24 21:40:49.196946 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.196715    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9" event={"ID":"77fd1550-ef43-41cc-8a2d-356ea250de44","Type":"ContainerDied","Data":"bde07b0a546b582ab2780482ee23cbeea70ab979d4f4d1c21c956621004576dc"}
Jul 24 21:40:49.196946 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.196738    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bde07b0a546b582ab2780482ee23cbeea70ab979d4f4d1c21c956621004576dc"
Jul 24 21:40:49.197244 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.197226    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:40:49.223386 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.223365    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77fd1550-ef43-41cc-8a2d-356ea250de44-kserve-provision-location\") pod \"77fd1550-ef43-41cc-8a2d-356ea250de44\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") "
Jul 24 21:40:49.223492 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.223396    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77fd1550-ef43-41cc-8a2d-356ea250de44-cabundle-cert\") pod \"77fd1550-ef43-41cc-8a2d-356ea250de44\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") "
Jul 24 21:40:49.223492 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.223467    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77fd1550-ef43-41cc-8a2d-356ea250de44-proxy-tls\") pod \"77fd1550-ef43-41cc-8a2d-356ea250de44\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") "
Jul 24 21:40:49.223608 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.223526    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hlnt\" (UniqueName: \"kubernetes.io/projected/77fd1550-ef43-41cc-8a2d-356ea250de44-kube-api-access-8hlnt\") pod \"77fd1550-ef43-41cc-8a2d-356ea250de44\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") "
Jul 24 21:40:49.223608 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.223560    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77fd1550-ef43-41cc-8a2d-356ea250de44-kube-rbac-proxy-sar-config\") pod \"77fd1550-ef43-41cc-8a2d-356ea250de44\" (UID: \"77fd1550-ef43-41cc-8a2d-356ea250de44\") "
Jul 24 21:40:49.223706 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.223645    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77fd1550-ef43-41cc-8a2d-356ea250de44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77fd1550-ef43-41cc-8a2d-356ea250de44" (UID: "77fd1550-ef43-41cc-8a2d-356ea250de44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:40:49.223815 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.223788    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fd1550-ef43-41cc-8a2d-356ea250de44-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "77fd1550-ef43-41cc-8a2d-356ea250de44" (UID: "77fd1550-ef43-41cc-8a2d-356ea250de44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:40:49.223929 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.223827    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77fd1550-ef43-41cc-8a2d-356ea250de44-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:40:49.223986 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.223946    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fd1550-ef43-41cc-8a2d-356ea250de44-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77fd1550-ef43-41cc-8a2d-356ea250de44" (UID: "77fd1550-ef43-41cc-8a2d-356ea250de44"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:40:49.225767 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.225708    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77fd1550-ef43-41cc-8a2d-356ea250de44-kube-api-access-8hlnt" (OuterVolumeSpecName: "kube-api-access-8hlnt") pod "77fd1550-ef43-41cc-8a2d-356ea250de44" (UID: "77fd1550-ef43-41cc-8a2d-356ea250de44"). InnerVolumeSpecName "kube-api-access-8hlnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:40:49.225767 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.225762    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77fd1550-ef43-41cc-8a2d-356ea250de44-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77fd1550-ef43-41cc-8a2d-356ea250de44" (UID: "77fd1550-ef43-41cc-8a2d-356ea250de44"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:40:49.324993 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.324964    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hlnt\" (UniqueName: \"kubernetes.io/projected/77fd1550-ef43-41cc-8a2d-356ea250de44-kube-api-access-8hlnt\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:40:49.324993 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.324988    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77fd1550-ef43-41cc-8a2d-356ea250de44-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:40:49.324993 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.324997    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77fd1550-ef43-41cc-8a2d-356ea250de44-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:40:49.325231 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.325006    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77fd1550-ef43-41cc-8a2d-356ea250de44-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:40:49.890313 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.890271    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:40:49.973625 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.973590    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:40:49.985313 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985289    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"]
Jul 24 21:40:49.985614 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985602    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kube-rbac-proxy"
Jul 24 21:40:49.985658 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985617    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kube-rbac-proxy"
Jul 24 21:40:49.985658 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985624    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kube-rbac-proxy"
Jul 24 21:40:49.985658 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985629    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kube-rbac-proxy"
Jul 24 21:40:49.985658 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985642    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="storage-initializer"
Jul 24 21:40:49.985658 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985649    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="storage-initializer"
Jul 24 21:40:49.985804 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985663    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kserve-container"
Jul 24 21:40:49.985804 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985668    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kserve-container"
Jul 24 21:40:49.985804 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985674    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="storage-initializer"
Jul 24 21:40:49.985804 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985680    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="storage-initializer"
Jul 24 21:40:49.985804 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985686    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kserve-container"
Jul 24 21:40:49.985804 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985690    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kserve-container"
Jul 24 21:40:49.985804 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985742    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kserve-container"
Jul 24 21:40:49.985804 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985751    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="609cfeac-e91b-4503-b8fa-d65d560a6281" containerName="kube-rbac-proxy"
Jul 24 21:40:49.985804 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985756    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kube-rbac-proxy"
Jul 24 21:40:49.985804 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.985762    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" containerName="kserve-container"
Jul 24 21:40:49.989084 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.989066    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:49.992000 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.991978    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 24 21:40:49.992255 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:49.992241    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 24 21:40:50.012899 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.012878    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"]
Jul 24 21:40:50.030311 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.030227    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.030311 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.030293    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.030477 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.030337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.030477 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.030391    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.030477 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.030434    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz4jj\" (UniqueName: \"kubernetes.io/projected/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kube-api-access-zz4jj\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.044999 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.044978    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:40:50.130835 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.130807    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.130957 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.130851    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.130957 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.130883    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.130957 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.130906    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz4jj\" (UniqueName: \"kubernetes.io/projected/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kube-api-access-zz4jj\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.130957 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.130926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.131184 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:40:50.130958    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 24 21:40:50.131184 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:40:50.131036    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-proxy-tls podName:429abbfa-0ca6-426e-bc0d-f088bb53ffa1 nodeName:}" failed. No retries permitted until 2026-07-24 21:40:50.631019805 +0000 UTC m=+1529.387365599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-proxy-tls") pod "isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" (UID: "429abbfa-0ca6-426e-bc0d-f088bb53ffa1") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 24 21:40:50.131391 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.131369    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.131663 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.131645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.131736 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.131680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.143876 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.143850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz4jj\" (UniqueName: \"kubernetes.io/projected/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kube-api-access-zz4jj\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.199252 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.199187    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:40:50.200422 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.200404    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"
Jul 24 21:40:50.219977 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.219949    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"]
Jul 24 21:40:50.226008 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.225986    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5ff9f946fc-8x6q9"]
Jul 24 21:40:50.635181 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.635149    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.637715 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.637683    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:50.899981 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:50.899885    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:51.031623 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:51.031553    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"]
Jul 24 21:40:51.033931 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:40:51.033906    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429abbfa_0ca6_426e_bc0d_f088bb53ffa1.slice/crio-8c83ad20e188b8aa06ccfb1c3c9dbb4d3bb3bbeb708dddd7a57b226ed60edd0e WatchSource:0}: Error finding container 8c83ad20e188b8aa06ccfb1c3c9dbb4d3bb3bbeb708dddd7a57b226ed60edd0e: Status 404 returned error can't find the container with id 8c83ad20e188b8aa06ccfb1c3c9dbb4d3bb3bbeb708dddd7a57b226ed60edd0e
Jul 24 21:40:51.207043 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:51.206946    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" event={"ID":"429abbfa-0ca6-426e-bc0d-f088bb53ffa1","Type":"ContainerStarted","Data":"b575c267199f1f610f30a6b32f0ff93b4bf3fd44cace40cc31da44ac0dfe7a94"}
Jul 24 21:40:51.207043 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:51.207022    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" event={"ID":"429abbfa-0ca6-426e-bc0d-f088bb53ffa1","Type":"ContainerStarted","Data":"8c83ad20e188b8aa06ccfb1c3c9dbb4d3bb3bbeb708dddd7a57b226ed60edd0e"}
Jul 24 21:40:51.753480 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:51.753449    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77fd1550-ef43-41cc-8a2d-356ea250de44" path="/var/lib/kubelet/pods/77fd1550-ef43-41cc-8a2d-356ea250de44/volumes"
Jul 24 21:40:55.225235 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:55.225202    2560 generic.go:356] "Generic (PLEG): container finished" podID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerID="b575c267199f1f610f30a6b32f0ff93b4bf3fd44cace40cc31da44ac0dfe7a94" exitCode=0
Jul 24 21:40:55.225663 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:55.225260    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" event={"ID":"429abbfa-0ca6-426e-bc0d-f088bb53ffa1","Type":"ContainerDied","Data":"b575c267199f1f610f30a6b32f0ff93b4bf3fd44cace40cc31da44ac0dfe7a94"}
Jul 24 21:40:56.231701 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:56.231667    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" event={"ID":"429abbfa-0ca6-426e-bc0d-f088bb53ffa1","Type":"ContainerStarted","Data":"454ec3de147b8c13792cdd7e15c4f692aae6c44e95593d20cba67954e07816de"}
Jul 24 21:40:56.232162 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:56.231727    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" event={"ID":"429abbfa-0ca6-426e-bc0d-f088bb53ffa1","Type":"ContainerStarted","Data":"86e2f4556734202b0c834399cf08cfe3e74539a291ff49c398dd4690d86118c8"}
Jul 24 21:40:56.232162 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:56.231988    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:56.232162 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:56.232031    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:40:56.233426 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:56.233402    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 24 21:40:56.251229 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:56.251187    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" podStartSLOduration=3.570370629 podStartE2EDuration="7.251173979s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.68080335s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:40:56.25002048 +0000 UTC m=+1535.006366293" watchObservedRunningTime="2026-07-24 21:40:56.251173979 +0000 UTC m=+1535.007519877"
Jul 24 21:40:57.236042 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:40:57.236007    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 24 21:41:02.240338 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:41:02.240303    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:41:02.240903 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:41:02.240877    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 24 21:41:12.241308 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:41:12.241262    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 24 21:41:22.241120 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:41:22.241081    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 24 21:41:32.240850 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:41:32.240803    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 24 21:41:42.241539 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:41:42.241499    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 24 21:41:52.241666 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:41:52.241626    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 24 21:42:02.241363 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:02.241319    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 24 21:42:12.242046 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:12.242008    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:42:20.025022 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.024989    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"]
Jul 24 21:42:20.025584 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.025233    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kserve-container" containerID="cri-o://86e2f4556734202b0c834399cf08cfe3e74539a291ff49c398dd4690d86118c8" gracePeriod=30
Jul 24 21:42:20.025584 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.025304    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kube-rbac-proxy" containerID="cri-o://454ec3de147b8c13792cdd7e15c4f692aae6c44e95593d20cba67954e07816de" gracePeriod=30
Jul 24 21:42:20.042632 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.042609    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:42:20.109349 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.109310    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"]
Jul 24 21:42:20.113076 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.113060    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.116302 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.116281    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 24 21:42:20.131425 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.131403    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"]
Jul 24 21:42:20.191720 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.191697    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-bqvb4\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.191861 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.191730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-bqvb4\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.191861 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.191776    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-proxy-tls\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-bqvb4\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.191861 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.191799    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d22zf\" (UniqueName: \"kubernetes.io/projected/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kube-api-access-d22zf\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-bqvb4\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.192008 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.191870    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-bqvb4\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.292928 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.292856    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-bqvb4\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.292928 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.292902    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-bqvb4\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.293112 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.293011    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-bqvb4\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.293112 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.293062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-proxy-tls\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-bqvb4\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.293112 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.293082    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d22zf\" (UniqueName: \"kubernetes.io/projected/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kube-api-access-d22zf\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-bqvb4\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.293303 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.293284    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-bqvb4\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.293668 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.293647    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-bqvb4\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.293710 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.293647    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-bqvb4\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.295526 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.295496    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-proxy-tls\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-bqvb4\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.301797 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.301777    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d22zf\" (UniqueName: \"kubernetes.io/projected/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kube-api-access-d22zf\") pod \"isvc-canary-rollback-predictor-6d775fdd5d-bqvb4\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.423494 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.423453    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:20.560966 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.560895    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"]
Jul 24 21:42:20.563944 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:42:20.563920    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ec258d_d47f_46c6_96b4_e09dec1bd75f.slice/crio-11cde3b584bb769729d2bd2175e06043d9dbf134d83faf439640345203ebe6e9 WatchSource:0}: Error finding container 11cde3b584bb769729d2bd2175e06043d9dbf134d83faf439640345203ebe6e9: Status 404 returned error can't find the container with id 11cde3b584bb769729d2bd2175e06043d9dbf134d83faf439640345203ebe6e9
Jul 24 21:42:20.565842 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.565821    2560 generic.go:356] "Generic (PLEG): container finished" podID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerID="454ec3de147b8c13792cdd7e15c4f692aae6c44e95593d20cba67954e07816de" exitCode=2
Jul 24 21:42:20.565913 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:20.565884    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" event={"ID":"429abbfa-0ca6-426e-bc0d-f088bb53ffa1","Type":"ContainerDied","Data":"454ec3de147b8c13792cdd7e15c4f692aae6c44e95593d20cba67954e07816de"}
Jul 24 21:42:21.571409 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:21.571372    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" event={"ID":"d6ec258d-d47f-46c6-96b4-e09dec1bd75f","Type":"ContainerStarted","Data":"dbdd9451eca60b83d1dde07fa0b4e5774a69dd6c8209c00827731776d0ab21db"}
Jul 24 21:42:21.571877 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:21.571445    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" event={"ID":"d6ec258d-d47f-46c6-96b4-e09dec1bd75f","Type":"ContainerStarted","Data":"11cde3b584bb769729d2bd2175e06043d9dbf134d83faf439640345203ebe6e9"}
Jul 24 21:42:22.236731 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:22.236684    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 24 21:42:22.241096 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:22.241066    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 24 21:42:24.586002 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:24.585966    2560 generic.go:356] "Generic (PLEG): container finished" podID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerID="dbdd9451eca60b83d1dde07fa0b4e5774a69dd6c8209c00827731776d0ab21db" exitCode=0
Jul 24 21:42:24.586422 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:24.586035    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" event={"ID":"d6ec258d-d47f-46c6-96b4-e09dec1bd75f","Type":"ContainerDied","Data":"dbdd9451eca60b83d1dde07fa0b4e5774a69dd6c8209c00827731776d0ab21db"}
Jul 24 21:42:24.881894 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:24.881869    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:42:24.925195 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:24.925161    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-cabundle-cert\") pod \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") "
Jul 24 21:42:24.925373 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:24.925259    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz4jj\" (UniqueName: \"kubernetes.io/projected/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kube-api-access-zz4jj\") pod \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") "
Jul 24 21:42:24.925373 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:24.925290    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kserve-provision-location\") pod \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") "
Jul 24 21:42:24.925373 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:24.925333    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kube-rbac-proxy-sar-config\") pod \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") "
Jul 24 21:42:24.925373 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:24.925368    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-proxy-tls\") pod \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\" (UID: \"429abbfa-0ca6-426e-bc0d-f088bb53ffa1\") "
Jul 24 21:42:24.925595 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:24.925555    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "429abbfa-0ca6-426e-bc0d-f088bb53ffa1" (UID: "429abbfa-0ca6-426e-bc0d-f088bb53ffa1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:42:24.925655 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:24.925580    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "429abbfa-0ca6-426e-bc0d-f088bb53ffa1" (UID: "429abbfa-0ca6-426e-bc0d-f088bb53ffa1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:42:24.925794 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:24.925760    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "429abbfa-0ca6-426e-bc0d-f088bb53ffa1" (UID: "429abbfa-0ca6-426e-bc0d-f088bb53ffa1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:42:24.927495 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:24.927474    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kube-api-access-zz4jj" (OuterVolumeSpecName: "kube-api-access-zz4jj") pod "429abbfa-0ca6-426e-bc0d-f088bb53ffa1" (UID: "429abbfa-0ca6-426e-bc0d-f088bb53ffa1"). InnerVolumeSpecName "kube-api-access-zz4jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:42:24.927564 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:24.927512    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "429abbfa-0ca6-426e-bc0d-f088bb53ffa1" (UID: "429abbfa-0ca6-426e-bc0d-f088bb53ffa1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:42:25.026565 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.026500    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:42:25.026565 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.026523    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:42:25.026565 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.026532    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:42:25.026565 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.026540    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz4jj\" (UniqueName: \"kubernetes.io/projected/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kube-api-access-zz4jj\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:42:25.026565 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.026550    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/429abbfa-0ca6-426e-bc0d-f088bb53ffa1-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:42:25.592300 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.592266    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" event={"ID":"d6ec258d-d47f-46c6-96b4-e09dec1bd75f","Type":"ContainerStarted","Data":"1e18bec878cb3c5138ec7892fdd8150e9a2aae4a6216a00132940e582c91e786"}
Jul 24 21:42:25.592773 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.592341    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" event={"ID":"d6ec258d-d47f-46c6-96b4-e09dec1bd75f","Type":"ContainerStarted","Data":"292f7450787ddc9c534647b86878ed38ba443512d6fc28ac6a45c564e3c60969"}
Jul 24 21:42:25.592773 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.592542    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:25.594165 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.594144    2560 generic.go:356] "Generic (PLEG): container finished" podID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerID="86e2f4556734202b0c834399cf08cfe3e74539a291ff49c398dd4690d86118c8" exitCode=0
Jul 24 21:42:25.594267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.594192    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"
Jul 24 21:42:25.594267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.594201    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" event={"ID":"429abbfa-0ca6-426e-bc0d-f088bb53ffa1","Type":"ContainerDied","Data":"86e2f4556734202b0c834399cf08cfe3e74539a291ff49c398dd4690d86118c8"}
Jul 24 21:42:25.594267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.594241    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm" event={"ID":"429abbfa-0ca6-426e-bc0d-f088bb53ffa1","Type":"ContainerDied","Data":"8c83ad20e188b8aa06ccfb1c3c9dbb4d3bb3bbeb708dddd7a57b226ed60edd0e"}
Jul 24 21:42:25.594267 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.594267    2560 scope.go:117] "RemoveContainer" containerID="454ec3de147b8c13792cdd7e15c4f692aae6c44e95593d20cba67954e07816de"
Jul 24 21:42:25.605675 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.605653    2560 scope.go:117] "RemoveContainer" containerID="86e2f4556734202b0c834399cf08cfe3e74539a291ff49c398dd4690d86118c8"
Jul 24 21:42:25.613582 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.613546    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" podStartSLOduration=1.773957415 podStartE2EDuration="5.613532383s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.839574968s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:42:25.612975598 +0000 UTC m=+1624.369321411" watchObservedRunningTime="2026-07-24 21:42:25.613532383 +0000 UTC m=+1624.369878259"
Jul 24 21:42:25.614265 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.614241    2560 scope.go:117] "RemoveContainer" containerID="b575c267199f1f610f30a6b32f0ff93b4bf3fd44cace40cc31da44ac0dfe7a94"
Jul 24 21:42:25.621852 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.621836    2560 scope.go:117] "RemoveContainer" containerID="454ec3de147b8c13792cdd7e15c4f692aae6c44e95593d20cba67954e07816de"
Jul 24 21:42:25.622116 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:42:25.622101    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454ec3de147b8c13792cdd7e15c4f692aae6c44e95593d20cba67954e07816de\": container with ID starting with 454ec3de147b8c13792cdd7e15c4f692aae6c44e95593d20cba67954e07816de not found: ID does not exist" containerID="454ec3de147b8c13792cdd7e15c4f692aae6c44e95593d20cba67954e07816de"
Jul 24 21:42:25.622178 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.622148    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454ec3de147b8c13792cdd7e15c4f692aae6c44e95593d20cba67954e07816de"} err="failed to get container status \"454ec3de147b8c13792cdd7e15c4f692aae6c44e95593d20cba67954e07816de\": rpc error: code = NotFound desc = could not find container \"454ec3de147b8c13792cdd7e15c4f692aae6c44e95593d20cba67954e07816de\": container with ID starting with 454ec3de147b8c13792cdd7e15c4f692aae6c44e95593d20cba67954e07816de not found: ID does not exist"
Jul 24 21:42:25.622178 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.622167    2560 scope.go:117] "RemoveContainer" containerID="86e2f4556734202b0c834399cf08cfe3e74539a291ff49c398dd4690d86118c8"
Jul 24 21:42:25.622389 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:42:25.622374    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e2f4556734202b0c834399cf08cfe3e74539a291ff49c398dd4690d86118c8\": container with ID starting with 86e2f4556734202b0c834399cf08cfe3e74539a291ff49c398dd4690d86118c8 not found: ID does not exist" containerID="86e2f4556734202b0c834399cf08cfe3e74539a291ff49c398dd4690d86118c8"
Jul 24 21:42:25.622428 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.622396    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e2f4556734202b0c834399cf08cfe3e74539a291ff49c398dd4690d86118c8"} err="failed to get container status \"86e2f4556734202b0c834399cf08cfe3e74539a291ff49c398dd4690d86118c8\": rpc error: code = NotFound desc = could not find container \"86e2f4556734202b0c834399cf08cfe3e74539a291ff49c398dd4690d86118c8\": container with ID starting with 86e2f4556734202b0c834399cf08cfe3e74539a291ff49c398dd4690d86118c8 not found: ID does not exist"
Jul 24 21:42:25.622428 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.622413    2560 scope.go:117] "RemoveContainer" containerID="b575c267199f1f610f30a6b32f0ff93b4bf3fd44cace40cc31da44ac0dfe7a94"
Jul 24 21:42:25.622612 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:42:25.622598    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b575c267199f1f610f30a6b32f0ff93b4bf3fd44cace40cc31da44ac0dfe7a94\": container with ID starting with b575c267199f1f610f30a6b32f0ff93b4bf3fd44cace40cc31da44ac0dfe7a94 not found: ID does not exist" containerID="b575c267199f1f610f30a6b32f0ff93b4bf3fd44cace40cc31da44ac0dfe7a94"
Jul 24 21:42:25.622658 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.622616    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b575c267199f1f610f30a6b32f0ff93b4bf3fd44cace40cc31da44ac0dfe7a94"} err="failed to get container status \"b575c267199f1f610f30a6b32f0ff93b4bf3fd44cace40cc31da44ac0dfe7a94\": rpc error: code = NotFound desc = could not find container \"b575c267199f1f610f30a6b32f0ff93b4bf3fd44cace40cc31da44ac0dfe7a94\": container with ID starting with b575c267199f1f610f30a6b32f0ff93b4bf3fd44cace40cc31da44ac0dfe7a94 not found: ID does not exist"
Jul 24 21:42:25.625927 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.625908    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"]
Jul 24 21:42:25.630663 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.630643    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5b9f7db4c5-jb5xm"]
Jul 24 21:42:25.753636 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:25.753556    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" path="/var/lib/kubelet/pods/429abbfa-0ca6-426e-bc0d-f088bb53ffa1/volumes"
Jul 24 21:42:26.599035 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:26.599000    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:26.600229 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:26.600202    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 24 21:42:27.602944 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:27.602904    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 24 21:42:32.607771 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:32.607745    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:42:32.608303 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:32.608275    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 24 21:42:42.609107 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:42.609065    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 24 21:42:52.608980 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:42:52.608938    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 24 21:43:02.608377 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:02.608339    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 24 21:43:12.608643 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:12.608601    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 24 21:43:22.609027 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:22.608993    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 24 21:43:32.609114 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:32.609065    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 24 21:43:36.749989 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:36.749959    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:43:40.183556 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:40.183528    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"]
Jul 24 21:43:40.184037 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:40.183761    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kserve-container" containerID="cri-o://292f7450787ddc9c534647b86878ed38ba443512d6fc28ac6a45c564e3c60969" gracePeriod=30
Jul 24 21:43:40.184037 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:40.183856    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kube-rbac-proxy" containerID="cri-o://1e18bec878cb3c5138ec7892fdd8150e9a2aae4a6216a00132940e582c91e786" gracePeriod=30
Jul 24 21:43:40.211005 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:40.210974    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:43:40.898269 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:40.898240    2560 generic.go:356] "Generic (PLEG): container finished" podID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerID="1e18bec878cb3c5138ec7892fdd8150e9a2aae4a6216a00132940e582c91e786" exitCode=2
Jul 24 21:43:40.898439 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:40.898305    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" event={"ID":"d6ec258d-d47f-46c6-96b4-e09dec1bd75f","Type":"ContainerDied","Data":"1e18bec878cb3c5138ec7892fdd8150e9a2aae4a6216a00132940e582c91e786"}
Jul 24 21:43:42.603695 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:42.603609    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 24 21:43:44.526909 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.526883    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:43:44.576017 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.575991    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-proxy-tls\") pod \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") "
Jul 24 21:43:44.576180 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.576058    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kube-rbac-proxy-sar-config\") pod \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") "
Jul 24 21:43:44.576180 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.576074    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-cabundle-cert\") pod \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") "
Jul 24 21:43:44.576180 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.576099    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d22zf\" (UniqueName: \"kubernetes.io/projected/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kube-api-access-d22zf\") pod \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") "
Jul 24 21:43:44.576180 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.576164    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kserve-provision-location\") pod \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\" (UID: \"d6ec258d-d47f-46c6-96b4-e09dec1bd75f\") "
Jul 24 21:43:44.576494 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.576466    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6ec258d-d47f-46c6-96b4-e09dec1bd75f" (UID: "d6ec258d-d47f-46c6-96b4-e09dec1bd75f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:43:44.576557 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.576511    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d6ec258d-d47f-46c6-96b4-e09dec1bd75f" (UID: "d6ec258d-d47f-46c6-96b4-e09dec1bd75f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:43:44.576557 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.576545    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6ec258d-d47f-46c6-96b4-e09dec1bd75f" (UID: "d6ec258d-d47f-46c6-96b4-e09dec1bd75f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:43:44.578377 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.578347    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kube-api-access-d22zf" (OuterVolumeSpecName: "kube-api-access-d22zf") pod "d6ec258d-d47f-46c6-96b4-e09dec1bd75f" (UID: "d6ec258d-d47f-46c6-96b4-e09dec1bd75f"). InnerVolumeSpecName "kube-api-access-d22zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:43:44.578377 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.578370    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6ec258d-d47f-46c6-96b4-e09dec1bd75f" (UID: "d6ec258d-d47f-46c6-96b4-e09dec1bd75f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:43:44.677211 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.677151    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d22zf\" (UniqueName: \"kubernetes.io/projected/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kube-api-access-d22zf\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:43:44.677211 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.677173    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:43:44.677211 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.677184    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:43:44.677211 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.677194    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:43:44.677211 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.677203    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6ec258d-d47f-46c6-96b4-e09dec1bd75f-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:43:44.916416 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.916381    2560 generic.go:356] "Generic (PLEG): container finished" podID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerID="292f7450787ddc9c534647b86878ed38ba443512d6fc28ac6a45c564e3c60969" exitCode=0
Jul 24 21:43:44.916596 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.916438    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"
Jul 24 21:43:44.916596 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.916495    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" event={"ID":"d6ec258d-d47f-46c6-96b4-e09dec1bd75f","Type":"ContainerDied","Data":"292f7450787ddc9c534647b86878ed38ba443512d6fc28ac6a45c564e3c60969"}
Jul 24 21:43:44.916596 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.916553    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4" event={"ID":"d6ec258d-d47f-46c6-96b4-e09dec1bd75f","Type":"ContainerDied","Data":"11cde3b584bb769729d2bd2175e06043d9dbf134d83faf439640345203ebe6e9"}
Jul 24 21:43:44.916596 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.916592    2560 scope.go:117] "RemoveContainer" containerID="1e18bec878cb3c5138ec7892fdd8150e9a2aae4a6216a00132940e582c91e786"
Jul 24 21:43:44.925644 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.925625    2560 scope.go:117] "RemoveContainer" containerID="292f7450787ddc9c534647b86878ed38ba443512d6fc28ac6a45c564e3c60969"
Jul 24 21:43:44.932892 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.932875    2560 scope.go:117] "RemoveContainer" containerID="dbdd9451eca60b83d1dde07fa0b4e5774a69dd6c8209c00827731776d0ab21db"
Jul 24 21:43:44.938668 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.938648    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"]
Jul 24 21:43:44.940900 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.940819    2560 scope.go:117] "RemoveContainer" containerID="1e18bec878cb3c5138ec7892fdd8150e9a2aae4a6216a00132940e582c91e786"
Jul 24 21:43:44.941270 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:43:44.941243    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e18bec878cb3c5138ec7892fdd8150e9a2aae4a6216a00132940e582c91e786\": container with ID starting with 1e18bec878cb3c5138ec7892fdd8150e9a2aae4a6216a00132940e582c91e786 not found: ID does not exist" containerID="1e18bec878cb3c5138ec7892fdd8150e9a2aae4a6216a00132940e582c91e786"
Jul 24 21:43:44.941413 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.941280    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e18bec878cb3c5138ec7892fdd8150e9a2aae4a6216a00132940e582c91e786"} err="failed to get container status \"1e18bec878cb3c5138ec7892fdd8150e9a2aae4a6216a00132940e582c91e786\": rpc error: code = NotFound desc = could not find container \"1e18bec878cb3c5138ec7892fdd8150e9a2aae4a6216a00132940e582c91e786\": container with ID starting with 1e18bec878cb3c5138ec7892fdd8150e9a2aae4a6216a00132940e582c91e786 not found: ID does not exist"
Jul 24 21:43:44.941413 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.941310    2560 scope.go:117] "RemoveContainer" containerID="292f7450787ddc9c534647b86878ed38ba443512d6fc28ac6a45c564e3c60969"
Jul 24 21:43:44.941727 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:43:44.941704    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292f7450787ddc9c534647b86878ed38ba443512d6fc28ac6a45c564e3c60969\": container with ID starting with 292f7450787ddc9c534647b86878ed38ba443512d6fc28ac6a45c564e3c60969 not found: ID does not exist" containerID="292f7450787ddc9c534647b86878ed38ba443512d6fc28ac6a45c564e3c60969"
Jul 24 21:43:44.941782 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.941733    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292f7450787ddc9c534647b86878ed38ba443512d6fc28ac6a45c564e3c60969"} err="failed to get container status \"292f7450787ddc9c534647b86878ed38ba443512d6fc28ac6a45c564e3c60969\": rpc error: code = NotFound desc = could not find container \"292f7450787ddc9c534647b86878ed38ba443512d6fc28ac6a45c564e3c60969\": container with ID starting with 292f7450787ddc9c534647b86878ed38ba443512d6fc28ac6a45c564e3c60969 not found: ID does not exist"
Jul 24 21:43:44.941782 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.941752    2560 scope.go:117] "RemoveContainer" containerID="dbdd9451eca60b83d1dde07fa0b4e5774a69dd6c8209c00827731776d0ab21db"
Jul 24 21:43:44.942013 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:43:44.941992    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbdd9451eca60b83d1dde07fa0b4e5774a69dd6c8209c00827731776d0ab21db\": container with ID starting with dbdd9451eca60b83d1dde07fa0b4e5774a69dd6c8209c00827731776d0ab21db not found: ID does not exist" containerID="dbdd9451eca60b83d1dde07fa0b4e5774a69dd6c8209c00827731776d0ab21db"
Jul 24 21:43:44.942057 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.942024    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbdd9451eca60b83d1dde07fa0b4e5774a69dd6c8209c00827731776d0ab21db"} err="failed to get container status \"dbdd9451eca60b83d1dde07fa0b4e5774a69dd6c8209c00827731776d0ab21db\": rpc error: code = NotFound desc = could not find container \"dbdd9451eca60b83d1dde07fa0b4e5774a69dd6c8209c00827731776d0ab21db\": container with ID starting with dbdd9451eca60b83d1dde07fa0b4e5774a69dd6c8209c00827731776d0ab21db not found: ID does not exist"
Jul 24 21:43:44.942941 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:44.942925    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6d775fdd5d-bqvb4"]
Jul 24 21:43:45.315188 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.315073    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:43:45.362466 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362439    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"]
Jul 24 21:43:45.362740 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362728    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="storage-initializer"
Jul 24 21:43:45.362794 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362741    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="storage-initializer"
Jul 24 21:43:45.362794 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362751    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kserve-container"
Jul 24 21:43:45.362794 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362756    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kserve-container"
Jul 24 21:43:45.362794 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362767    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kserve-container"
Jul 24 21:43:45.362794 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362772    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kserve-container"
Jul 24 21:43:45.362794 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362783    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="storage-initializer"
Jul 24 21:43:45.362794 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362788    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="storage-initializer"
Jul 24 21:43:45.363039 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362800    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kube-rbac-proxy"
Jul 24 21:43:45.363039 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362805    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kube-rbac-proxy"
Jul 24 21:43:45.363039 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362809    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kube-rbac-proxy"
Jul 24 21:43:45.363039 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362814    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kube-rbac-proxy"
Jul 24 21:43:45.363039 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362872    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kube-rbac-proxy"
Jul 24 21:43:45.363039 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362881    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="429abbfa-0ca6-426e-bc0d-f088bb53ffa1" containerName="kserve-container"
Jul 24 21:43:45.363039 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362887    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kserve-container"
Jul 24 21:43:45.363039 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.362893    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" containerName="kube-rbac-proxy"
Jul 24 21:43:45.367629 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.367609    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:45.370067 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.370046    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:43:45.370196 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.370069    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 24 21:43:45.370253 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.370079    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 24 21:43:45.370457 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.370443    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:43:45.370504 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.370460    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dm6jh\""
Jul 24 21:43:45.370544 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.370517    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:43:45.370583 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.370517    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:43:45.378286 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.378263    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"]
Jul 24 21:43:45.381461 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.381432    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:43:45.483691 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.483661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e59ad05-6266-4547-baec-0a5cca60c4ba-proxy-tls\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:45.483846 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.483703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e59ad05-6266-4547-baec-0a5cca60c4ba-cabundle-cert\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:45.483846 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.483750    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e59ad05-6266-4547-baec-0a5cca60c4ba-kserve-provision-location\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:45.483846 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.483773    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkpl9\" (UniqueName: \"kubernetes.io/projected/5e59ad05-6266-4547-baec-0a5cca60c4ba-kube-api-access-kkpl9\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:45.483846 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.483812    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e59ad05-6266-4547-baec-0a5cca60c4ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:45.585327 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.585194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkpl9\" (UniqueName: \"kubernetes.io/projected/5e59ad05-6266-4547-baec-0a5cca60c4ba-kube-api-access-kkpl9\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:45.585717 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.585351    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e59ad05-6266-4547-baec-0a5cca60c4ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:45.585717 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.585382    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e59ad05-6266-4547-baec-0a5cca60c4ba-proxy-tls\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:45.585717 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.585419    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e59ad05-6266-4547-baec-0a5cca60c4ba-cabundle-cert\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:45.585717 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.585467    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e59ad05-6266-4547-baec-0a5cca60c4ba-kserve-provision-location\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:45.585717 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:43:45.585539    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 24 21:43:45.585717 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:43:45.585638    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e59ad05-6266-4547-baec-0a5cca60c4ba-proxy-tls podName:5e59ad05-6266-4547-baec-0a5cca60c4ba nodeName:}" failed. No retries permitted until 2026-07-24 21:43:46.085617639 +0000 UTC m=+1704.841963430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5e59ad05-6266-4547-baec-0a5cca60c4ba-proxy-tls") pod "isvc-canary-stop-predictor-8bd7d86fd-gwc6t" (UID: "5e59ad05-6266-4547-baec-0a5cca60c4ba") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 24 21:43:45.585941 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.585785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e59ad05-6266-4547-baec-0a5cca60c4ba-kserve-provision-location\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:45.586009 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.585990    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e59ad05-6266-4547-baec-0a5cca60c4ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:45.586183 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.586168    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e59ad05-6266-4547-baec-0a5cca60c4ba-cabundle-cert\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:45.594035 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.594016    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkpl9\" (UniqueName: \"kubernetes.io/projected/5e59ad05-6266-4547-baec-0a5cca60c4ba-kube-api-access-kkpl9\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:45.748474 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.748440    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:43:45.753955 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.753927    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6ec258d-d47f-46c6-96b4-e09dec1bd75f" path="/var/lib/kubelet/pods/d6ec258d-d47f-46c6-96b4-e09dec1bd75f/volumes"
Jul 24 21:43:45.849500 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:45.849431    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:43:46.090385 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:46.090362    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e59ad05-6266-4547-baec-0a5cca60c4ba-proxy-tls\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:46.092651 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:46.092635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e59ad05-6266-4547-baec-0a5cca60c4ba-proxy-tls\") pod \"isvc-canary-stop-predictor-8bd7d86fd-gwc6t\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:46.278942 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:46.278914    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:46.417396 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:46.417366    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"]
Jul 24 21:43:46.420531 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:43:46.420505    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e59ad05_6266_4547_baec_0a5cca60c4ba.slice/crio-42490d6f1a464fac3e1dd8d6bbda4753574771b10b94142117c3a739ee357841 WatchSource:0}: Error finding container 42490d6f1a464fac3e1dd8d6bbda4753574771b10b94142117c3a739ee357841: Status 404 returned error can't find the container with id 42490d6f1a464fac3e1dd8d6bbda4753574771b10b94142117c3a739ee357841
Jul 24 21:43:46.926831 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:46.926798    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" event={"ID":"5e59ad05-6266-4547-baec-0a5cca60c4ba","Type":"ContainerStarted","Data":"6cda0a008a4b14ddb07687d0b908c9d152aaadd9b793bfe599becf4e642f23b4"}
Jul 24 21:43:46.927305 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:46.926857    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" event={"ID":"5e59ad05-6266-4547-baec-0a5cca60c4ba","Type":"ContainerStarted","Data":"42490d6f1a464fac3e1dd8d6bbda4753574771b10b94142117c3a739ee357841"}
Jul 24 21:43:50.944871 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:50.944839    2560 generic.go:356] "Generic (PLEG): container finished" podID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerID="6cda0a008a4b14ddb07687d0b908c9d152aaadd9b793bfe599becf4e642f23b4" exitCode=0
Jul 24 21:43:50.945300 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:50.944911    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" event={"ID":"5e59ad05-6266-4547-baec-0a5cca60c4ba","Type":"ContainerDied","Data":"6cda0a008a4b14ddb07687d0b908c9d152aaadd9b793bfe599becf4e642f23b4"}
Jul 24 21:43:51.953800 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:51.953767    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" event={"ID":"5e59ad05-6266-4547-baec-0a5cca60c4ba","Type":"ContainerStarted","Data":"768c16e7754da698755a57c6fdee73e0e7145125ac26a7417c06a877a6318290"}
Jul 24 21:43:51.954190 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:51.953827    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" event={"ID":"5e59ad05-6266-4547-baec-0a5cca60c4ba","Type":"ContainerStarted","Data":"c1e8974d4354697faffc4ed1648a061ac50959cb3ec2c83f28f642211de423e0"}
Jul 24 21:43:51.954190 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:51.954062    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:51.973817 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:51.973762    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" podStartSLOduration=2.967915786 podStartE2EDuration="6.973747029s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.005831243s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:43:51.972095477 +0000 UTC m=+1710.728441332" watchObservedRunningTime="2026-07-24 21:43:51.973747029 +0000 UTC m=+1710.730092842"
Jul 24 21:43:52.958081 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:52.958050    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:52.959398 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:52.959364    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:43:53.962356 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:53.962309    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:43:58.966891 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:58.966862    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:43:58.967326 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:43:58.967300    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:44:08.967619 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:44:08.967576    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:44:18.967356 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:44:18.967319    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:44:21.818408 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:44:21.818377    2560 scope.go:117] "RemoveContainer" containerID="aa1f7924c934efa8f997f92592760f82da251f6271232d3d57e680a57f8ec95c"
Jul 24 21:44:21.826207 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:44:21.826189    2560 scope.go:117] "RemoveContainer" containerID="eacc7ba6197bf166268436444f085ec9dff52783fa830a85efdac5d2376c8c51"
Jul 24 21:44:21.833663 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:44:21.833646    2560 scope.go:117] "RemoveContainer" containerID="f275fdd96de1caf63904ccf03fd0954932d22fa7b09a7d0de1682e1f945b2d47"
Jul 24 21:44:28.967722 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:44:28.967683    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:44:38.968240 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:44:38.968200    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:44:48.968107 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:44:48.968070    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:44:58.967814 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:44:58.967776    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:45:08.967940 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:08.967912    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:45:15.408339 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:15.408306    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"]
Jul 24 21:45:15.408889 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:15.408574    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kserve-container" containerID="cri-o://c1e8974d4354697faffc4ed1648a061ac50959cb3ec2c83f28f642211de423e0" gracePeriod=30
Jul 24 21:45:15.408889 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:15.408578    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kube-rbac-proxy" containerID="cri-o://768c16e7754da698755a57c6fdee73e0e7145125ac26a7417c06a877a6318290" gracePeriod=30
Jul 24 21:45:15.416695 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:15.416667    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:45:15.518809 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:15.518768    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:45:16.282906 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:16.282873    2560 generic.go:356] "Generic (PLEG): container finished" podID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerID="768c16e7754da698755a57c6fdee73e0e7145125ac26a7417c06a877a6318290" exitCode=2
Jul 24 21:45:16.283084 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:16.282925    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" event={"ID":"5e59ad05-6266-4547-baec-0a5cca60c4ba","Type":"ContainerDied","Data":"768c16e7754da698755a57c6fdee73e0e7145125ac26a7417c06a877a6318290"}
Jul 24 21:45:18.962945 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:18.962900    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 24 21:45:18.967290 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:18.967261    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 24 21:45:19.957289 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:19.957267    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:45:20.057459 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.057384    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkpl9\" (UniqueName: \"kubernetes.io/projected/5e59ad05-6266-4547-baec-0a5cca60c4ba-kube-api-access-kkpl9\") pod \"5e59ad05-6266-4547-baec-0a5cca60c4ba\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") "
Jul 24 21:45:20.057891 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.057461    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e59ad05-6266-4547-baec-0a5cca60c4ba-cabundle-cert\") pod \"5e59ad05-6266-4547-baec-0a5cca60c4ba\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") "
Jul 24 21:45:20.057891 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.057506    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e59ad05-6266-4547-baec-0a5cca60c4ba-proxy-tls\") pod \"5e59ad05-6266-4547-baec-0a5cca60c4ba\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") "
Jul 24 21:45:20.057891 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.057554    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e59ad05-6266-4547-baec-0a5cca60c4ba-kserve-provision-location\") pod \"5e59ad05-6266-4547-baec-0a5cca60c4ba\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") "
Jul 24 21:45:20.057891 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.057577    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e59ad05-6266-4547-baec-0a5cca60c4ba-kube-rbac-proxy-sar-config\") pod \"5e59ad05-6266-4547-baec-0a5cca60c4ba\" (UID: \"5e59ad05-6266-4547-baec-0a5cca60c4ba\") "
Jul 24 21:45:20.057891 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.057855    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e59ad05-6266-4547-baec-0a5cca60c4ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e59ad05-6266-4547-baec-0a5cca60c4ba" (UID: "5e59ad05-6266-4547-baec-0a5cca60c4ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:45:20.058103 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.057841    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e59ad05-6266-4547-baec-0a5cca60c4ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5e59ad05-6266-4547-baec-0a5cca60c4ba" (UID: "5e59ad05-6266-4547-baec-0a5cca60c4ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:45:20.058103 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.057978    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e59ad05-6266-4547-baec-0a5cca60c4ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5e59ad05-6266-4547-baec-0a5cca60c4ba" (UID: "5e59ad05-6266-4547-baec-0a5cca60c4ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:45:20.059689 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.059665    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e59ad05-6266-4547-baec-0a5cca60c4ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5e59ad05-6266-4547-baec-0a5cca60c4ba" (UID: "5e59ad05-6266-4547-baec-0a5cca60c4ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:45:20.059689 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.059675    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e59ad05-6266-4547-baec-0a5cca60c4ba-kube-api-access-kkpl9" (OuterVolumeSpecName: "kube-api-access-kkpl9") pod "5e59ad05-6266-4547-baec-0a5cca60c4ba" (UID: "5e59ad05-6266-4547-baec-0a5cca60c4ba"). InnerVolumeSpecName "kube-api-access-kkpl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:45:20.159069 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.159040    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e59ad05-6266-4547-baec-0a5cca60c4ba-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:45:20.159069 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.159064    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e59ad05-6266-4547-baec-0a5cca60c4ba-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:45:20.159069 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.159072    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e59ad05-6266-4547-baec-0a5cca60c4ba-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:45:20.159289 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.159082    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e59ad05-6266-4547-baec-0a5cca60c4ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:45:20.159289 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.159091    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkpl9\" (UniqueName: \"kubernetes.io/projected/5e59ad05-6266-4547-baec-0a5cca60c4ba-kube-api-access-kkpl9\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:45:20.300718 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.300681    2560 generic.go:356] "Generic (PLEG): container finished" podID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerID="c1e8974d4354697faffc4ed1648a061ac50959cb3ec2c83f28f642211de423e0" exitCode=0
Jul 24 21:45:20.300874 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.300730    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"
Jul 24 21:45:20.300874 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.300770    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" event={"ID":"5e59ad05-6266-4547-baec-0a5cca60c4ba","Type":"ContainerDied","Data":"c1e8974d4354697faffc4ed1648a061ac50959cb3ec2c83f28f642211de423e0"}
Jul 24 21:45:20.300874 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.300830    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t" event={"ID":"5e59ad05-6266-4547-baec-0a5cca60c4ba","Type":"ContainerDied","Data":"42490d6f1a464fac3e1dd8d6bbda4753574771b10b94142117c3a739ee357841"}
Jul 24 21:45:20.300874 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.300857    2560 scope.go:117] "RemoveContainer" containerID="768c16e7754da698755a57c6fdee73e0e7145125ac26a7417c06a877a6318290"
Jul 24 21:45:20.309851 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.309834    2560 scope.go:117] "RemoveContainer" containerID="c1e8974d4354697faffc4ed1648a061ac50959cb3ec2c83f28f642211de423e0"
Jul 24 21:45:20.317460 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.317446    2560 scope.go:117] "RemoveContainer" containerID="6cda0a008a4b14ddb07687d0b908c9d152aaadd9b793bfe599becf4e642f23b4"
Jul 24 21:45:20.323334 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.323315    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"]
Jul 24 21:45:20.324891 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.324872    2560 scope.go:117] "RemoveContainer" containerID="768c16e7754da698755a57c6fdee73e0e7145125ac26a7417c06a877a6318290"
Jul 24 21:45:20.325191 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:45:20.325165    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768c16e7754da698755a57c6fdee73e0e7145125ac26a7417c06a877a6318290\": container with ID starting with 768c16e7754da698755a57c6fdee73e0e7145125ac26a7417c06a877a6318290 not found: ID does not exist" containerID="768c16e7754da698755a57c6fdee73e0e7145125ac26a7417c06a877a6318290"
Jul 24 21:45:20.325295 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.325199    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768c16e7754da698755a57c6fdee73e0e7145125ac26a7417c06a877a6318290"} err="failed to get container status \"768c16e7754da698755a57c6fdee73e0e7145125ac26a7417c06a877a6318290\": rpc error: code = NotFound desc = could not find container \"768c16e7754da698755a57c6fdee73e0e7145125ac26a7417c06a877a6318290\": container with ID starting with 768c16e7754da698755a57c6fdee73e0e7145125ac26a7417c06a877a6318290 not found: ID does not exist"
Jul 24 21:45:20.325295 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.325222    2560 scope.go:117] "RemoveContainer" containerID="c1e8974d4354697faffc4ed1648a061ac50959cb3ec2c83f28f642211de423e0"
Jul 24 21:45:20.325519 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:45:20.325480    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e8974d4354697faffc4ed1648a061ac50959cb3ec2c83f28f642211de423e0\": container with ID starting with c1e8974d4354697faffc4ed1648a061ac50959cb3ec2c83f28f642211de423e0 not found: ID does not exist" containerID="c1e8974d4354697faffc4ed1648a061ac50959cb3ec2c83f28f642211de423e0"
Jul 24 21:45:20.325618 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.325527    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e8974d4354697faffc4ed1648a061ac50959cb3ec2c83f28f642211de423e0"} err="failed to get container status \"c1e8974d4354697faffc4ed1648a061ac50959cb3ec2c83f28f642211de423e0\": rpc error: code = NotFound desc = could not find container \"c1e8974d4354697faffc4ed1648a061ac50959cb3ec2c83f28f642211de423e0\": container with ID starting with c1e8974d4354697faffc4ed1648a061ac50959cb3ec2c83f28f642211de423e0 not found: ID does not exist"
Jul 24 21:45:20.325618 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.325552    2560 scope.go:117] "RemoveContainer" containerID="6cda0a008a4b14ddb07687d0b908c9d152aaadd9b793bfe599becf4e642f23b4"
Jul 24 21:45:20.325973 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:45:20.325904    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cda0a008a4b14ddb07687d0b908c9d152aaadd9b793bfe599becf4e642f23b4\": container with ID starting with 6cda0a008a4b14ddb07687d0b908c9d152aaadd9b793bfe599becf4e642f23b4 not found: ID does not exist" containerID="6cda0a008a4b14ddb07687d0b908c9d152aaadd9b793bfe599becf4e642f23b4"
Jul 24 21:45:20.325973 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.325947    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cda0a008a4b14ddb07687d0b908c9d152aaadd9b793bfe599becf4e642f23b4"} err="failed to get container status \"6cda0a008a4b14ddb07687d0b908c9d152aaadd9b793bfe599becf4e642f23b4\": rpc error: code = NotFound desc = could not find container \"6cda0a008a4b14ddb07687d0b908c9d152aaadd9b793bfe599becf4e642f23b4\": container with ID starting with 6cda0a008a4b14ddb07687d0b908c9d152aaadd9b793bfe599becf4e642f23b4 not found: ID does not exist"
Jul 24 21:45:20.327578 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:20.327557    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-8bd7d86fd-gwc6t"]
Jul 24 21:45:21.748504 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:21.748475    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:45:21.749079 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:21.748780    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:45:21.758789 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:21.758761    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" path="/var/lib/kubelet/pods/5e59ad05-6266-4547-baec-0a5cca60c4ba/volumes"
Jul 24 21:45:25.690960 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:25.690923    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:45:26.151162 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:45:26.151115    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:00.929188 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.929092    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:00.960050 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.960021    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"]
Jul 24 21:47:00.960377 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.960352    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kube-rbac-proxy"
Jul 24 21:47:00.960424 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.960379    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kube-rbac-proxy"
Jul 24 21:47:00.960424 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.960389    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="storage-initializer"
Jul 24 21:47:00.960424 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.960394    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="storage-initializer"
Jul 24 21:47:00.960424 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.960408    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kserve-container"
Jul 24 21:47:00.960424 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.960414    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kserve-container"
Jul 24 21:47:00.960576 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.960463    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kube-rbac-proxy"
Jul 24 21:47:00.960576 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.960474    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e59ad05-6266-4547-baec-0a5cca60c4ba" containerName="kserve-container"
Jul 24 21:47:00.963555 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.963538    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:00.966227 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.966201    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-cbae98-predictor-serving-cert\""
Jul 24 21:47:00.966344 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.966233    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:47:00.966344 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.966202    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-cbae98-kube-rbac-proxy-sar-config\""
Jul 24 21:47:00.966344 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.966202    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-cbae98\""
Jul 24 21:47:00.966344 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.966208    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-cbae98-dockercfg-k2lmw\""
Jul 24 21:47:00.966344 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.966202    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:47:00.966913 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.966898    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:47:00.977030 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:00.977003    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"]
Jul 24 21:47:01.065913 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.065883    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:01.071111 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.071076    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kserve-provision-location\") pod \"isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") " pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.071254 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.071160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nltlq\" (UniqueName: \"kubernetes.io/projected/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kube-api-access-nltlq\") pod \"isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") " pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.071254 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.071192    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-cabundle-cert\") pod \"isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") " pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.071350 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.071261    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-proxy-tls\") pod \"isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") " pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.071350 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.071306    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") " pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.172237 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.172204    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-proxy-tls\") pod \"isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") " pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.172395 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.172244    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") " pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.172395 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.172282    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kserve-provision-location\") pod \"isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") " pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.172526 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.172385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nltlq\" (UniqueName: \"kubernetes.io/projected/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kube-api-access-nltlq\") pod \"isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") " pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.172526 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.172428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-cabundle-cert\") pod \"isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") " pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.172704 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.172685    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kserve-provision-location\") pod \"isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") " pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.173024 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.173004    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") " pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.173081 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.173064    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-cabundle-cert\") pod \"isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") " pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.174768 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.174747    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-proxy-tls\") pod \"isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") " pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.181532 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.181473    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nltlq\" (UniqueName: \"kubernetes.io/projected/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kube-api-access-nltlq\") pod \"isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") " pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.273360 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.273323    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:01.411291 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.411266    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"]
Jul 24 21:47:01.413774 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:47:01.413745    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf3d8ad_1248_4e38_8984_c1499c3e71f8.slice/crio-df605bca2e3fba77482394d7f1ee4445a7b7843fda224f5e6fc2b801d743c473 WatchSource:0}: Error finding container df605bca2e3fba77482394d7f1ee4445a7b7843fda224f5e6fc2b801d743c473: Status 404 returned error can't find the container with id df605bca2e3fba77482394d7f1ee4445a7b7843fda224f5e6fc2b801d743c473
Jul 24 21:47:01.690999 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.690961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r" event={"ID":"dcf3d8ad-1248-4e38-8984-c1499c3e71f8","Type":"ContainerStarted","Data":"1e3bafb1ab575e3f640349f905163288d3aefdf144811e5cb67d0cc178bce3cb"}
Jul 24 21:47:01.691199 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:01.691046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r" event={"ID":"dcf3d8ad-1248-4e38-8984-c1499c3e71f8","Type":"ContainerStarted","Data":"df605bca2e3fba77482394d7f1ee4445a7b7843fda224f5e6fc2b801d743c473"}
Jul 24 21:47:06.714027 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:06.714001    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r_dcf3d8ad-1248-4e38-8984-c1499c3e71f8/storage-initializer/0.log"
Jul 24 21:47:06.714450 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:06.714037    2560 generic.go:356] "Generic (PLEG): container finished" podID="dcf3d8ad-1248-4e38-8984-c1499c3e71f8" containerID="1e3bafb1ab575e3f640349f905163288d3aefdf144811e5cb67d0cc178bce3cb" exitCode=1
Jul 24 21:47:06.714450 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:06.714117    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r" event={"ID":"dcf3d8ad-1248-4e38-8984-c1499c3e71f8","Type":"ContainerDied","Data":"1e3bafb1ab575e3f640349f905163288d3aefdf144811e5cb67d0cc178bce3cb"}
Jul 24 21:47:07.720292 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:07.720259    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r_dcf3d8ad-1248-4e38-8984-c1499c3e71f8/storage-initializer/0.log"
Jul 24 21:47:07.720694 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:07.720355    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r" event={"ID":"dcf3d8ad-1248-4e38-8984-c1499c3e71f8","Type":"ContainerStarted","Data":"7d2b6ece3394fecbba1bd541e174ebdb0c5f57273e53c3a930d677b38d3c9565"}
Jul 24 21:47:12.743292 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:12.743263    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r_dcf3d8ad-1248-4e38-8984-c1499c3e71f8/storage-initializer/1.log"
Jul 24 21:47:12.743684 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:12.743589    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r_dcf3d8ad-1248-4e38-8984-c1499c3e71f8/storage-initializer/0.log"
Jul 24 21:47:12.743684 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:12.743618    2560 generic.go:356] "Generic (PLEG): container finished" podID="dcf3d8ad-1248-4e38-8984-c1499c3e71f8" containerID="7d2b6ece3394fecbba1bd541e174ebdb0c5f57273e53c3a930d677b38d3c9565" exitCode=1
Jul 24 21:47:12.743684 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:12.743667    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r" event={"ID":"dcf3d8ad-1248-4e38-8984-c1499c3e71f8","Type":"ContainerDied","Data":"7d2b6ece3394fecbba1bd541e174ebdb0c5f57273e53c3a930d677b38d3c9565"}
Jul 24 21:47:12.743788 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:12.743744    2560 scope.go:117] "RemoveContainer" containerID="1e3bafb1ab575e3f640349f905163288d3aefdf144811e5cb67d0cc178bce3cb"
Jul 24 21:47:12.744066 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:12.744037    2560 scope.go:117] "RemoveContainer" containerID="1e3bafb1ab575e3f640349f905163288d3aefdf144811e5cb67d0cc178bce3cb"
Jul 24 21:47:12.754745 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:47:12.754723    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r_kserve-ci-e2e-test(dcf3d8ad-1248-4e38-8984-c1499c3e71f8)\"" pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r" podUID="dcf3d8ad-1248-4e38-8984-c1499c3e71f8"
Jul 24 21:47:12.755249 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:47:12.755228    2560 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r_kserve-ci-e2e-test_dcf3d8ad-1248-4e38-8984-c1499c3e71f8_0 in pod sandbox df605bca2e3fba77482394d7f1ee4445a7b7843fda224f5e6fc2b801d743c473 from index: no such id: '1e3bafb1ab575e3f640349f905163288d3aefdf144811e5cb67d0cc178bce3cb'" containerID="1e3bafb1ab575e3f640349f905163288d3aefdf144811e5cb67d0cc178bce3cb"
Jul 24 21:47:12.755316 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:12.755259    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3bafb1ab575e3f640349f905163288d3aefdf144811e5cb67d0cc178bce3cb"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r_kserve-ci-e2e-test_dcf3d8ad-1248-4e38-8984-c1499c3e71f8_0 in pod sandbox df605bca2e3fba77482394d7f1ee4445a7b7843fda224f5e6fc2b801d743c473 from index: no such id: '1e3bafb1ab575e3f640349f905163288d3aefdf144811e5cb67d0cc178bce3cb'"
Jul 24 21:47:13.748674 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:13.748645    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r_dcf3d8ad-1248-4e38-8984-c1499c3e71f8/storage-initializer/1.log"
Jul 24 21:47:16.989329 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:16.989286    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"]
Jul 24 21:47:17.015021 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.014985    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:17.091308 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.091277    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:17.128824 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.128805    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r_dcf3d8ad-1248-4e38-8984-c1499c3e71f8/storage-initializer/1.log"
Jul 24 21:47:17.132276 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.132256    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:17.194965 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.194947    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nltlq\" (UniqueName: \"kubernetes.io/projected/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kube-api-access-nltlq\") pod \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") "
Jul 24 21:47:17.195078 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.194975    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-cabundle-cert\") pod \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") "
Jul 24 21:47:17.195078 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.195006    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kube-rbac-proxy-sar-config\") pod \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") "
Jul 24 21:47:17.195078 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.195045    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-proxy-tls\") pod \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") "
Jul 24 21:47:17.195078 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.195069    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kserve-provision-location\") pod \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\" (UID: \"dcf3d8ad-1248-4e38-8984-c1499c3e71f8\") "
Jul 24 21:47:17.195425 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.195398    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dcf3d8ad-1248-4e38-8984-c1499c3e71f8" (UID: "dcf3d8ad-1248-4e38-8984-c1499c3e71f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:47:17.195533 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.195443    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dcf3d8ad-1248-4e38-8984-c1499c3e71f8" (UID: "dcf3d8ad-1248-4e38-8984-c1499c3e71f8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:47:17.195533 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.195454    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dcf3d8ad-1248-4e38-8984-c1499c3e71f8" (UID: "dcf3d8ad-1248-4e38-8984-c1499c3e71f8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:47:17.197189 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.197168    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kube-api-access-nltlq" (OuterVolumeSpecName: "kube-api-access-nltlq") pod "dcf3d8ad-1248-4e38-8984-c1499c3e71f8" (UID: "dcf3d8ad-1248-4e38-8984-c1499c3e71f8"). InnerVolumeSpecName "kube-api-access-nltlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:47:17.197284 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.197187    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dcf3d8ad-1248-4e38-8984-c1499c3e71f8" (UID: "dcf3d8ad-1248-4e38-8984-c1499c3e71f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:47:17.295883 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.295816    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:47:17.295883 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.295841    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:47:17.295883 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.295851    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nltlq\" (UniqueName: \"kubernetes.io/projected/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kube-api-access-nltlq\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:47:17.295883 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.295861    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:47:17.295883 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.295869    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dcf3d8ad-1248-4e38-8984-c1499c3e71f8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:47:17.765106 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.765083    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r_dcf3d8ad-1248-4e38-8984-c1499c3e71f8/storage-initializer/1.log"
Jul 24 21:47:17.765250 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.765203    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"
Jul 24 21:47:17.765250 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.765226    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r" event={"ID":"dcf3d8ad-1248-4e38-8984-c1499c3e71f8","Type":"ContainerDied","Data":"df605bca2e3fba77482394d7f1ee4445a7b7843fda224f5e6fc2b801d743c473"}
Jul 24 21:47:17.765329 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.765291    2560 scope.go:117] "RemoveContainer" containerID="7d2b6ece3394fecbba1bd541e174ebdb0c5f57273e53c3a930d677b38d3c9565"
Jul 24 21:47:17.798809 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.798772    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"]
Jul 24 21:47:17.808521 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:17.808492    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-cbae98-predictor-6dcc6c88bf-wt44r"]
Jul 24 21:47:19.753090 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:19.753056    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcf3d8ad-1248-4e38-8984-c1499c3e71f8" path="/var/lib/kubelet/pods/dcf3d8ad-1248-4e38-8984-c1499c3e71f8/volumes"
Jul 24 21:47:22.190688 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:22.190603    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:22.328371 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:22.328339    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:32.268338 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:32.268296    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:37.430034 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.430003    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:37.461961 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.461930    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"]
Jul 24 21:47:37.462282 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.462268    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf3d8ad-1248-4e38-8984-c1499c3e71f8" containerName="storage-initializer"
Jul 24 21:47:37.462332 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.462285    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf3d8ad-1248-4e38-8984-c1499c3e71f8" containerName="storage-initializer"
Jul 24 21:47:37.462332 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.462297    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf3d8ad-1248-4e38-8984-c1499c3e71f8" containerName="storage-initializer"
Jul 24 21:47:37.462332 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.462304    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf3d8ad-1248-4e38-8984-c1499c3e71f8" containerName="storage-initializer"
Jul 24 21:47:37.462441 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.462360    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf3d8ad-1248-4e38-8984-c1499c3e71f8" containerName="storage-initializer"
Jul 24 21:47:37.462441 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.462367    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf3d8ad-1248-4e38-8984-c1499c3e71f8" containerName="storage-initializer"
Jul 24 21:47:37.465356 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.465339    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:37.467716 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.467695    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-1170d-predictor-serving-cert\""
Jul 24 21:47:37.467821 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.467714    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dm6jh\""
Jul 24 21:47:37.467821 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.467722    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 24 21:47:37.467821 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.467789    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-1170d-kube-rbac-proxy-sar-config\""
Jul 24 21:47:37.468046 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.468031    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 24 21:47:37.468089 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.468048    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 24 21:47:37.468505 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.468491    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 24 21:47:37.476222 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.476199    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"]
Jul 24 21:47:37.562469 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.562441    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/123ce881-597b-4bcf-a455-10bcb7bcc059-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:37.562631 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.562473    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc5f7\" (UniqueName: \"kubernetes.io/projected/123ce881-597b-4bcf-a455-10bcb7bcc059-kube-api-access-mc5f7\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:37.562631 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.562500    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/123ce881-597b-4bcf-a455-10bcb7bcc059-proxy-tls\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:37.562631 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.562576    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/123ce881-597b-4bcf-a455-10bcb7bcc059-cabundle-cert\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:37.562733 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.562643    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/123ce881-597b-4bcf-a455-10bcb7bcc059-kserve-provision-location\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:37.663923 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.663877    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/123ce881-597b-4bcf-a455-10bcb7bcc059-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:37.663923 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.663925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc5f7\" (UniqueName: \"kubernetes.io/projected/123ce881-597b-4bcf-a455-10bcb7bcc059-kube-api-access-mc5f7\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:37.664169 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.663956    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/123ce881-597b-4bcf-a455-10bcb7bcc059-proxy-tls\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:37.664169 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.663986    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/123ce881-597b-4bcf-a455-10bcb7bcc059-cabundle-cert\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:37.664169 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.664047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/123ce881-597b-4bcf-a455-10bcb7bcc059-kserve-provision-location\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:37.664169 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:47:37.664151    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-1170d-predictor-serving-cert: secret "raw-sklearn-1170d-predictor-serving-cert" not found
Jul 24 21:47:37.664347 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:47:37.664241    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123ce881-597b-4bcf-a455-10bcb7bcc059-proxy-tls podName:123ce881-597b-4bcf-a455-10bcb7bcc059 nodeName:}" failed. No retries permitted until 2026-07-24 21:47:38.164219287 +0000 UTC m=+1936.920565078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/123ce881-597b-4bcf-a455-10bcb7bcc059-proxy-tls") pod "raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" (UID: "123ce881-597b-4bcf-a455-10bcb7bcc059") : secret "raw-sklearn-1170d-predictor-serving-cert" not found
Jul 24 21:47:37.664489 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.664467    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/123ce881-597b-4bcf-a455-10bcb7bcc059-kserve-provision-location\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:37.664634 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.664616    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/123ce881-597b-4bcf-a455-10bcb7bcc059-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:37.664670 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.664635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/123ce881-597b-4bcf-a455-10bcb7bcc059-cabundle-cert\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:37.672692 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.672672    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc5f7\" (UniqueName: \"kubernetes.io/projected/123ce881-597b-4bcf-a455-10bcb7bcc059-kube-api-access-mc5f7\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:37.779322 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:37.779225    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:38.166908 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:38.166874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/123ce881-597b-4bcf-a455-10bcb7bcc059-proxy-tls\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:38.169421 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:38.169402    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/123ce881-597b-4bcf-a455-10bcb7bcc059-proxy-tls\") pod \"raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") " pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:38.376185 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:38.376154    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:38.508857 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:38.508789    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"]
Jul 24 21:47:38.511586 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:47:38.511562    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123ce881_597b_4bcf_a455_10bcb7bcc059.slice/crio-0ed66d0a184a1b1bc01bed44d8b0426216bd3f855dc074d4275049de9611eb8f WatchSource:0}: Error finding container 0ed66d0a184a1b1bc01bed44d8b0426216bd3f855dc074d4275049de9611eb8f: Status 404 returned error can't find the container with id 0ed66d0a184a1b1bc01bed44d8b0426216bd3f855dc074d4275049de9611eb8f
Jul 24 21:47:38.842635 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:38.842538    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" event={"ID":"123ce881-597b-4bcf-a455-10bcb7bcc059","Type":"ContainerStarted","Data":"b4bd037990636e5f47ebe6f8d7cb670e2977f04bba30943c4d6ead60c6ea90f8"}
Jul 24 21:47:38.842807 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:38.842651    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" event={"ID":"123ce881-597b-4bcf-a455-10bcb7bcc059","Type":"ContainerStarted","Data":"0ed66d0a184a1b1bc01bed44d8b0426216bd3f855dc074d4275049de9611eb8f"}
Jul 24 21:47:42.858529 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:42.858503    2560 generic.go:356] "Generic (PLEG): container finished" podID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerID="b4bd037990636e5f47ebe6f8d7cb670e2977f04bba30943c4d6ead60c6ea90f8" exitCode=0
Jul 24 21:47:42.858898 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:42.858574    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" event={"ID":"123ce881-597b-4bcf-a455-10bcb7bcc059","Type":"ContainerDied","Data":"b4bd037990636e5f47ebe6f8d7cb670e2977f04bba30943c4d6ead60c6ea90f8"}
Jul 24 21:47:43.864402 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:43.864368    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" event={"ID":"123ce881-597b-4bcf-a455-10bcb7bcc059","Type":"ContainerStarted","Data":"fd3dc4e86d80e20ca2c40ddcc416467cca3568aaf17d9120540d0b51980a716c"}
Jul 24 21:47:43.864847 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:43.864427    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" event={"ID":"123ce881-597b-4bcf-a455-10bcb7bcc059","Type":"ContainerStarted","Data":"f19a4159d42348eded1f2d776eed05119f1b3b039886a3c51dca53997f457b7c"}
Jul 24 21:47:43.864847 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:43.864763    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:43.864972 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:43.864889    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:43.866316 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:43.866290    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:47:43.899504 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:43.899447    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" podStartSLOduration=3.072866662 podStartE2EDuration="6.899432893s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.826566231s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:47:43.89849424 +0000 UTC m=+1942.654840055" watchObservedRunningTime="2026-07-24 21:47:43.899432893 +0000 UTC m=+1942.655778708"
Jul 24 21:47:44.867962 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:44.867923    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:47:49.872890 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:49.872862    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:47:49.873451 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:49.873428    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:47:58.256371 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:58.256336    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:47:59.873385 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:47:59.873344    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:48:09.874407 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:48:09.874363    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:48:19.873707 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:48:19.873662    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:48:29.873431 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:48:29.873386    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:48:39.873376 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:48:39.873331    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:48:49.873706 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:48:49.873665    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:48:59.874284 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:48:59.874256    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:49:07.632102 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:07.632072    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"]
Jul 24 21:49:07.632603 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:07.632454    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kserve-container" containerID="cri-o://f19a4159d42348eded1f2d776eed05119f1b3b039886a3c51dca53997f457b7c" gracePeriod=30
Jul 24 21:49:07.632603 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:07.632504    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kube-rbac-proxy" containerID="cri-o://fd3dc4e86d80e20ca2c40ddcc416467cca3568aaf17d9120540d0b51980a716c" gracePeriod=30
Jul 24 21:49:07.647160 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:07.647106    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:49:08.188096 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:08.188061    2560 generic.go:356] "Generic (PLEG): container finished" podID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerID="fd3dc4e86d80e20ca2c40ddcc416467cca3568aaf17d9120540d0b51980a716c" exitCode=2
Jul 24 21:49:08.188279 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:08.188146    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" event={"ID":"123ce881-597b-4bcf-a455-10bcb7bcc059","Type":"ContainerDied","Data":"fd3dc4e86d80e20ca2c40ddcc416467cca3568aaf17d9120540d0b51980a716c"}
Jul 24 21:49:09.868884 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:09.868837    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 24 21:49:09.874312 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:09.874281    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 24 21:49:12.379555 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.379530    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:49:12.546579 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.546495    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc5f7\" (UniqueName: \"kubernetes.io/projected/123ce881-597b-4bcf-a455-10bcb7bcc059-kube-api-access-mc5f7\") pod \"123ce881-597b-4bcf-a455-10bcb7bcc059\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") "
Jul 24 21:49:12.546579 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.546541    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/123ce881-597b-4bcf-a455-10bcb7bcc059-cabundle-cert\") pod \"123ce881-597b-4bcf-a455-10bcb7bcc059\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") "
Jul 24 21:49:12.546810 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.546585    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/123ce881-597b-4bcf-a455-10bcb7bcc059-proxy-tls\") pod \"123ce881-597b-4bcf-a455-10bcb7bcc059\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") "
Jul 24 21:49:12.546810 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.546620    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/123ce881-597b-4bcf-a455-10bcb7bcc059-kserve-provision-location\") pod \"123ce881-597b-4bcf-a455-10bcb7bcc059\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") "
Jul 24 21:49:12.546810 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.546647    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/123ce881-597b-4bcf-a455-10bcb7bcc059-kube-rbac-proxy-sar-config\") pod \"123ce881-597b-4bcf-a455-10bcb7bcc059\" (UID: \"123ce881-597b-4bcf-a455-10bcb7bcc059\") "
Jul 24 21:49:12.546974 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.546948    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123ce881-597b-4bcf-a455-10bcb7bcc059-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "123ce881-597b-4bcf-a455-10bcb7bcc059" (UID: "123ce881-597b-4bcf-a455-10bcb7bcc059"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:49:12.547015 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.546998    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123ce881-597b-4bcf-a455-10bcb7bcc059-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "123ce881-597b-4bcf-a455-10bcb7bcc059" (UID: "123ce881-597b-4bcf-a455-10bcb7bcc059"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:49:12.547050 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.547030    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123ce881-597b-4bcf-a455-10bcb7bcc059-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "123ce881-597b-4bcf-a455-10bcb7bcc059" (UID: "123ce881-597b-4bcf-a455-10bcb7bcc059"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:49:12.548855 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.548827    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123ce881-597b-4bcf-a455-10bcb7bcc059-kube-api-access-mc5f7" (OuterVolumeSpecName: "kube-api-access-mc5f7") pod "123ce881-597b-4bcf-a455-10bcb7bcc059" (UID: "123ce881-597b-4bcf-a455-10bcb7bcc059"). InnerVolumeSpecName "kube-api-access-mc5f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:49:12.548940 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.548900    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/123ce881-597b-4bcf-a455-10bcb7bcc059-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "123ce881-597b-4bcf-a455-10bcb7bcc059" (UID: "123ce881-597b-4bcf-a455-10bcb7bcc059"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:49:12.647479 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.647454    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/123ce881-597b-4bcf-a455-10bcb7bcc059-kserve-provision-location\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:49:12.647479 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.647475    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/123ce881-597b-4bcf-a455-10bcb7bcc059-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:49:12.647630 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.647484    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc5f7\" (UniqueName: \"kubernetes.io/projected/123ce881-597b-4bcf-a455-10bcb7bcc059-kube-api-access-mc5f7\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:49:12.647630 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.647493    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/123ce881-597b-4bcf-a455-10bcb7bcc059-cabundle-cert\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:49:12.647630 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.647503    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/123ce881-597b-4bcf-a455-10bcb7bcc059-proxy-tls\") on node \"ip-10-0-136-232.ec2.internal\" DevicePath \"\""
Jul 24 21:49:12.751317 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.751284    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:49:12.892189 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:12.892158    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:49:13.208440 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.208353    2560 generic.go:356] "Generic (PLEG): container finished" podID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerID="f19a4159d42348eded1f2d776eed05119f1b3b039886a3c51dca53997f457b7c" exitCode=0
Jul 24 21:49:13.208440 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.208406    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" event={"ID":"123ce881-597b-4bcf-a455-10bcb7bcc059","Type":"ContainerDied","Data":"f19a4159d42348eded1f2d776eed05119f1b3b039886a3c51dca53997f457b7c"}
Jul 24 21:49:13.208440 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.208417    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"
Jul 24 21:49:13.208670 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.208471    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5" event={"ID":"123ce881-597b-4bcf-a455-10bcb7bcc059","Type":"ContainerDied","Data":"0ed66d0a184a1b1bc01bed44d8b0426216bd3f855dc074d4275049de9611eb8f"}
Jul 24 21:49:13.208670 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.208514    2560 scope.go:117] "RemoveContainer" containerID="fd3dc4e86d80e20ca2c40ddcc416467cca3568aaf17d9120540d0b51980a716c"
Jul 24 21:49:13.217485 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.217467    2560 scope.go:117] "RemoveContainer" containerID="f19a4159d42348eded1f2d776eed05119f1b3b039886a3c51dca53997f457b7c"
Jul 24 21:49:13.224869 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.224848    2560 scope.go:117] "RemoveContainer" containerID="b4bd037990636e5f47ebe6f8d7cb670e2977f04bba30943c4d6ead60c6ea90f8"
Jul 24 21:49:13.232222 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.232151    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"]
Jul 24 21:49:13.232298 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.232285    2560 scope.go:117] "RemoveContainer" containerID="fd3dc4e86d80e20ca2c40ddcc416467cca3568aaf17d9120540d0b51980a716c"
Jul 24 21:49:13.232521 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:49:13.232504    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3dc4e86d80e20ca2c40ddcc416467cca3568aaf17d9120540d0b51980a716c\": container with ID starting with fd3dc4e86d80e20ca2c40ddcc416467cca3568aaf17d9120540d0b51980a716c not found: ID does not exist" containerID="fd3dc4e86d80e20ca2c40ddcc416467cca3568aaf17d9120540d0b51980a716c"
Jul 24 21:49:13.232563 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.232529    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3dc4e86d80e20ca2c40ddcc416467cca3568aaf17d9120540d0b51980a716c"} err="failed to get container status \"fd3dc4e86d80e20ca2c40ddcc416467cca3568aaf17d9120540d0b51980a716c\": rpc error: code = NotFound desc = could not find container \"fd3dc4e86d80e20ca2c40ddcc416467cca3568aaf17d9120540d0b51980a716c\": container with ID starting with fd3dc4e86d80e20ca2c40ddcc416467cca3568aaf17d9120540d0b51980a716c not found: ID does not exist"
Jul 24 21:49:13.232563 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.232546    2560 scope.go:117] "RemoveContainer" containerID="f19a4159d42348eded1f2d776eed05119f1b3b039886a3c51dca53997f457b7c"
Jul 24 21:49:13.232780 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:49:13.232759    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19a4159d42348eded1f2d776eed05119f1b3b039886a3c51dca53997f457b7c\": container with ID starting with f19a4159d42348eded1f2d776eed05119f1b3b039886a3c51dca53997f457b7c not found: ID does not exist" containerID="f19a4159d42348eded1f2d776eed05119f1b3b039886a3c51dca53997f457b7c"
Jul 24 21:49:13.232824 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.232785    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19a4159d42348eded1f2d776eed05119f1b3b039886a3c51dca53997f457b7c"} err="failed to get container status \"f19a4159d42348eded1f2d776eed05119f1b3b039886a3c51dca53997f457b7c\": rpc error: code = NotFound desc = could not find container \"f19a4159d42348eded1f2d776eed05119f1b3b039886a3c51dca53997f457b7c\": container with ID starting with f19a4159d42348eded1f2d776eed05119f1b3b039886a3c51dca53997f457b7c not found: ID does not exist"
Jul 24 21:49:13.232824 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.232801    2560 scope.go:117] "RemoveContainer" containerID="b4bd037990636e5f47ebe6f8d7cb670e2977f04bba30943c4d6ead60c6ea90f8"
Jul 24 21:49:13.233022 ip-10-0-136-232 kubenswrapper[2560]: E0724 21:49:13.233007    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4bd037990636e5f47ebe6f8d7cb670e2977f04bba30943c4d6ead60c6ea90f8\": container with ID starting with b4bd037990636e5f47ebe6f8d7cb670e2977f04bba30943c4d6ead60c6ea90f8 not found: ID does not exist" containerID="b4bd037990636e5f47ebe6f8d7cb670e2977f04bba30943c4d6ead60c6ea90f8"
Jul 24 21:49:13.233085 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.233024    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4bd037990636e5f47ebe6f8d7cb670e2977f04bba30943c4d6ead60c6ea90f8"} err="failed to get container status \"b4bd037990636e5f47ebe6f8d7cb670e2977f04bba30943c4d6ead60c6ea90f8\": rpc error: code = NotFound desc = could not find container \"b4bd037990636e5f47ebe6f8d7cb670e2977f04bba30943c4d6ead60c6ea90f8\": container with ID starting with b4bd037990636e5f47ebe6f8d7cb670e2977f04bba30943c4d6ead60c6ea90f8 not found: ID does not exist"
Jul 24 21:49:13.236734 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.236716    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-1170d-predictor-7b76dc8fff-bkbz5"]
Jul 24 21:49:13.753669 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:49:13.753637    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" path="/var/lib/kubelet/pods/123ce881-597b-4bcf-a455-10bcb7bcc059/volumes"
Jul 24 21:50:17.398476 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:50:17.398437    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:50:21.772775 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:50:21.772745    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:50:21.773672 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:50:21.773654    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:50:42.945690 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:50:42.945652    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:50:49.044886 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:50:49.044851    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:50:49.146482 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:50:49.146455    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:50:56.302978 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:50:56.302937    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:51:19.021048 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.021015    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:51:19.176988 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.176955    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-64p2h/must-gather-lcpgz"]
Jul 24 21:51:19.177285 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.177271    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="storage-initializer"
Jul 24 21:51:19.177335 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.177288    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="storage-initializer"
Jul 24 21:51:19.177335 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.177312    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kserve-container"
Jul 24 21:51:19.177335 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.177320    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kserve-container"
Jul 24 21:51:19.177335 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.177331    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kube-rbac-proxy"
Jul 24 21:51:19.177459 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.177338    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kube-rbac-proxy"
Jul 24 21:51:19.177459 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.177391    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kube-rbac-proxy"
Jul 24 21:51:19.177459 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.177404    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="123ce881-597b-4bcf-a455-10bcb7bcc059" containerName="kserve-container"
Jul 24 21:51:19.180456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.180432    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-64p2h/must-gather-lcpgz"
Jul 24 21:51:19.183112 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.183092    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-64p2h\"/\"openshift-service-ca.crt\""
Jul 24 21:51:19.183890 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.183872    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-64p2h\"/\"kube-root-ca.crt\""
Jul 24 21:51:19.183981 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.183964    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-64p2h\"/\"default-dockercfg-7x6wv\""
Jul 24 21:51:19.189016 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.188995    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-64p2h/must-gather-lcpgz"]
Jul 24 21:51:19.192010 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.191987    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:51:19.255997 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.251668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b3618de7-f8df-4d62-bb0b-aac9d464bce1-must-gather-output\") pod \"must-gather-lcpgz\" (UID: \"b3618de7-f8df-4d62-bb0b-aac9d464bce1\") " pod="openshift-must-gather-64p2h/must-gather-lcpgz"
Jul 24 21:51:19.255997 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.251736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xff7\" (UniqueName: \"kubernetes.io/projected/b3618de7-f8df-4d62-bb0b-aac9d464bce1-kube-api-access-4xff7\") pod \"must-gather-lcpgz\" (UID: \"b3618de7-f8df-4d62-bb0b-aac9d464bce1\") " pod="openshift-must-gather-64p2h/must-gather-lcpgz"
Jul 24 21:51:19.292434 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.292365    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:51:19.352704 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.352672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b3618de7-f8df-4d62-bb0b-aac9d464bce1-must-gather-output\") pod \"must-gather-lcpgz\" (UID: \"b3618de7-f8df-4d62-bb0b-aac9d464bce1\") " pod="openshift-must-gather-64p2h/must-gather-lcpgz"
Jul 24 21:51:19.352839 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.352730    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xff7\" (UniqueName: \"kubernetes.io/projected/b3618de7-f8df-4d62-bb0b-aac9d464bce1-kube-api-access-4xff7\") pod \"must-gather-lcpgz\" (UID: \"b3618de7-f8df-4d62-bb0b-aac9d464bce1\") " pod="openshift-must-gather-64p2h/must-gather-lcpgz"
Jul 24 21:51:19.352999 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.352983    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b3618de7-f8df-4d62-bb0b-aac9d464bce1-must-gather-output\") pod \"must-gather-lcpgz\" (UID: \"b3618de7-f8df-4d62-bb0b-aac9d464bce1\") " pod="openshift-must-gather-64p2h/must-gather-lcpgz"
Jul 24 21:51:19.361544 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.361522    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xff7\" (UniqueName: \"kubernetes.io/projected/b3618de7-f8df-4d62-bb0b-aac9d464bce1-kube-api-access-4xff7\") pod \"must-gather-lcpgz\" (UID: \"b3618de7-f8df-4d62-bb0b-aac9d464bce1\") " pod="openshift-must-gather-64p2h/must-gather-lcpgz"
Jul 24 21:51:19.499443 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.499418    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-64p2h/must-gather-lcpgz"
Jul 24 21:51:19.628463 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.628434    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-64p2h/must-gather-lcpgz"]
Jul 24 21:51:19.631986 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:51:19.631960    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3618de7_f8df_4d62_bb0b_aac9d464bce1.slice/crio-19aac24d66e26103ddf80e0284cde68eb4137ee14db2a3c411e0ad412688fa1f WatchSource:0}: Error finding container 19aac24d66e26103ddf80e0284cde68eb4137ee14db2a3c411e0ad412688fa1f: Status 404 returned error can't find the container with id 19aac24d66e26103ddf80e0284cde68eb4137ee14db2a3c411e0ad412688fa1f
Jul 24 21:51:19.633845 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.633825    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 21:51:19.679531 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:19.679505    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-64p2h/must-gather-lcpgz" event={"ID":"b3618de7-f8df-4d62-bb0b-aac9d464bce1","Type":"ContainerStarted","Data":"19aac24d66e26103ddf80e0284cde68eb4137ee14db2a3c411e0ad412688fa1f"}
Jul 24 21:51:20.687504 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:20.687417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-64p2h/must-gather-lcpgz" event={"ID":"b3618de7-f8df-4d62-bb0b-aac9d464bce1","Type":"ContainerStarted","Data":"ef8f157c8318214fff0a711631c5268f1b4a307deab3db3cd84c768962c799e2"}
Jul 24 21:51:20.687504 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:20.687462    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-64p2h/must-gather-lcpgz" event={"ID":"b3618de7-f8df-4d62-bb0b-aac9d464bce1","Type":"ContainerStarted","Data":"921c89a403f98889fa8a8d6f30eef06f1a4638fda3643dff3437b5332e4e5734"}
Jul 24 21:51:20.706022 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:20.705971    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-64p2h/must-gather-lcpgz" podStartSLOduration=0.958359413 podStartE2EDuration="1.70595048s" totalImagesPullingTime="747.591067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:51:20.70493082 +0000 UTC m=+2159.461276645" watchObservedRunningTime="2026-07-24 21:51:20.70595048 +0000 UTC m=+2159.462296294"
Jul 24 21:51:21.842656 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:21.842629    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b868q_1ec4ede0-f187-4d6c-a8a2-dd853f431b01/global-pull-secret-syncer/0.log"
Jul 24 21:51:21.939601 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:21.939578    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log"
Jul 24 21:51:22.028963 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:22.028921    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-232.ec2.internal_e035e960d86bd45119eab5cda9361ecd/haproxy/0.log"
Jul 24 21:51:24.720338 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:24.720244    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:51:24.783004 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:24.782963    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:51:26.052787 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:26.052757    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-vlf8v_c944d753-b675-4664-944a-3ec3c10b224f/cluster-monitoring-operator/0.log"
Jul 24 21:51:26.086571 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:26.086543    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-mq6zw_dd2f032e-b417-4b44-8208-c885789525df/kube-state-metrics/0.log"
Jul 24 21:51:26.108881 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:26.108854    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-mq6zw_dd2f032e-b417-4b44-8208-c885789525df/kube-rbac-proxy-main/0.log"
Jul 24 21:51:26.136045 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:26.136013    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-mq6zw_dd2f032e-b417-4b44-8208-c885789525df/kube-rbac-proxy-self/0.log"
Jul 24 21:51:26.173465 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:26.173397    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-54d7c77989-5jbd8_700a1c63-ead1-4c2d-a32f-81070e959ceb/metrics-server/0.log"
Jul 24 21:51:26.332680 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:26.332605    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qq55j_14bf2715-d0e7-48ed-a231-2d6a1737f9bb/node-exporter/0.log"
Jul 24 21:51:26.368192 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:26.368119    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qq55j_14bf2715-d0e7-48ed-a231-2d6a1737f9bb/kube-rbac-proxy/0.log"
Jul 24 21:51:26.405018 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:26.404986    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qq55j_14bf2715-d0e7-48ed-a231-2d6a1737f9bb/init-textfile/0.log"
Jul 24 21:51:26.656977 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:26.656946    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_01482372-8206-4e25-943b-e98cf84a2c7b/prometheus/0.log"
Jul 24 21:51:26.707315 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:26.707265    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_01482372-8206-4e25-943b-e98cf84a2c7b/config-reloader/0.log"
Jul 24 21:51:26.751767 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:26.751740    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_01482372-8206-4e25-943b-e98cf84a2c7b/thanos-sidecar/0.log"
Jul 24 21:51:26.794257 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:26.794229    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_01482372-8206-4e25-943b-e98cf84a2c7b/kube-rbac-proxy-web/0.log"
Jul 24 21:51:26.840673 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:26.840625    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_01482372-8206-4e25-943b-e98cf84a2c7b/kube-rbac-proxy/0.log"
Jul 24 21:51:26.894021 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:26.893993    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_01482372-8206-4e25-943b-e98cf84a2c7b/kube-rbac-proxy-thanos/0.log"
Jul 24 21:51:26.955949 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:26.955882    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_01482372-8206-4e25-943b-e98cf84a2c7b/init-config-reloader/0.log"
Jul 24 21:51:27.033371 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:27.033343    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-2glc5_9330697e-c2ed-4687-a8db-8c99ed55b0a8/prometheus-operator/0.log"
Jul 24 21:51:27.110321 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:27.110296    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-2glc5_9330697e-c2ed-4687-a8db-8c99ed55b0a8/kube-rbac-proxy/0.log"
Jul 24 21:51:27.206342 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:27.206218    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-926h2_9ff33b94-17d1-4fd2-a256-6479f169b8b7/prometheus-operator-admission-webhook/0.log"
Jul 24 21:51:29.872924 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:29.872891    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"]
Jul 24 21:51:29.878598 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:29.878573    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:29.887378 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:29.887348    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"]
Jul 24 21:51:29.955152 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:29.954810    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9630fd7f-31cd-491c-b0b2-d0fe0a0661ed-podres\") pod \"perf-node-gather-daemonset-8njtr\" (UID: \"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed\") " pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:29.955152 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:29.954903    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9630fd7f-31cd-491c-b0b2-d0fe0a0661ed-proc\") pod \"perf-node-gather-daemonset-8njtr\" (UID: \"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed\") " pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:29.955152 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:29.954974    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9630fd7f-31cd-491c-b0b2-d0fe0a0661ed-lib-modules\") pod \"perf-node-gather-daemonset-8njtr\" (UID: \"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed\") " pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:29.955152 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:29.955081    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tvqg\" (UniqueName: \"kubernetes.io/projected/9630fd7f-31cd-491c-b0b2-d0fe0a0661ed-kube-api-access-5tvqg\") pod \"perf-node-gather-daemonset-8njtr\" (UID: \"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed\") " pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:29.955152 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:29.955151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9630fd7f-31cd-491c-b0b2-d0fe0a0661ed-sys\") pod \"perf-node-gather-daemonset-8njtr\" (UID: \"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed\") " pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:30.055648 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.055606    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tvqg\" (UniqueName: \"kubernetes.io/projected/9630fd7f-31cd-491c-b0b2-d0fe0a0661ed-kube-api-access-5tvqg\") pod \"perf-node-gather-daemonset-8njtr\" (UID: \"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed\") " pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:30.055849 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.055657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9630fd7f-31cd-491c-b0b2-d0fe0a0661ed-sys\") pod \"perf-node-gather-daemonset-8njtr\" (UID: \"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed\") " pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:30.055849 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.055696    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9630fd7f-31cd-491c-b0b2-d0fe0a0661ed-podres\") pod \"perf-node-gather-daemonset-8njtr\" (UID: \"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed\") " pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:30.055849 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.055747    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9630fd7f-31cd-491c-b0b2-d0fe0a0661ed-proc\") pod \"perf-node-gather-daemonset-8njtr\" (UID: \"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed\") " pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:30.055849 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.055801    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9630fd7f-31cd-491c-b0b2-d0fe0a0661ed-lib-modules\") pod \"perf-node-gather-daemonset-8njtr\" (UID: \"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed\") " pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:30.055849 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.055808    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9630fd7f-31cd-491c-b0b2-d0fe0a0661ed-sys\") pod \"perf-node-gather-daemonset-8njtr\" (UID: \"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed\") " pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:30.056154 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.055893    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9630fd7f-31cd-491c-b0b2-d0fe0a0661ed-podres\") pod \"perf-node-gather-daemonset-8njtr\" (UID: \"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed\") " pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:30.056154 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.055890    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9630fd7f-31cd-491c-b0b2-d0fe0a0661ed-proc\") pod \"perf-node-gather-daemonset-8njtr\" (UID: \"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed\") " pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:30.056154 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.055955    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9630fd7f-31cd-491c-b0b2-d0fe0a0661ed-lib-modules\") pod \"perf-node-gather-daemonset-8njtr\" (UID: \"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed\") " pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:30.064087 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.064057    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tvqg\" (UniqueName: \"kubernetes.io/projected/9630fd7f-31cd-491c-b0b2-d0fe0a0661ed-kube-api-access-5tvqg\") pod \"perf-node-gather-daemonset-8njtr\" (UID: \"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed\") " pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:30.194336 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.194245    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:30.363550 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.361824    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"]
Jul 24 21:51:30.365634 ip-10-0-136-232 kubenswrapper[2560]: W0724 21:51:30.365603    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9630fd7f_31cd_491c_b0b2_d0fe0a0661ed.slice/crio-311b64f67bc7252b96d3033d13a3d2368cb29e89215dcc48feb0c9c97c69d498 WatchSource:0}: Error finding container 311b64f67bc7252b96d3033d13a3d2368cb29e89215dcc48feb0c9c97c69d498: Status 404 returned error can't find the container with id 311b64f67bc7252b96d3033d13a3d2368cb29e89215dcc48feb0c9c97c69d498
Jul 24 21:51:30.436971 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.436947    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-ppqph_bb8acf81-3fb0-4095-9ae4-b9b514ffffe1/volume-data-source-validator/0.log"
Jul 24 21:51:30.744883 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.744786    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr" event={"ID":"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed","Type":"ContainerStarted","Data":"ef26d5858eef8cb755dfd8f7755a679310a8063545cca970af4d65c8b71c9e31"}
Jul 24 21:51:30.744883 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.744860    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr" event={"ID":"9630fd7f-31cd-491c-b0b2-d0fe0a0661ed","Type":"ContainerStarted","Data":"311b64f67bc7252b96d3033d13a3d2368cb29e89215dcc48feb0c9c97c69d498"}
Jul 24 21:51:30.745119 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.744971    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:30.764035 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:30.763976    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr" podStartSLOduration=1.76395674 podStartE2EDuration="1.76395674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:51:30.762936851 +0000 UTC m=+2169.519282675" watchObservedRunningTime="2026-07-24 21:51:30.76395674 +0000 UTC m=+2169.520302543"
Jul 24 21:51:31.519671 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:31.519641    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j8cqc_b0ade7d3-1225-4c63-8282-4b0ea5537005/dns/0.log"
Jul 24 21:51:31.563745 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:31.563717    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j8cqc_b0ade7d3-1225-4c63-8282-4b0ea5537005/kube-rbac-proxy/0.log"
Jul 24 21:51:31.674694 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:31.674663    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hgz77_62728d7d-6f8d-43dd-9824-67708026c9ff/dns-node-resolver/0.log"
Jul 24 21:51:32.226860 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:32.226821    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p5ggk_cfd5b405-1f77-48ea-a94f-c30b74583745/node-ca/0.log"
Jul 24 21:51:33.474674 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:33.474644    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hzssv_fb92e8b5-ef73-45bc-b383-718ecad57138/serve-healthcheck-canary/0.log"
Jul 24 21:51:34.242291 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:34.242258    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r75fx_d8999bbe-7e48-4678-8951-b232a39f1de9/kube-rbac-proxy/0.log"
Jul 24 21:51:34.276989 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:34.276958    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r75fx_d8999bbe-7e48-4678-8951-b232a39f1de9/exporter/0.log"
Jul 24 21:51:34.319670 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:34.319649    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r75fx_d8999bbe-7e48-4678-8951-b232a39f1de9/extractor/0.log"
Jul 24 21:51:36.761604 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:36.761574    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-64p2h/perf-node-gather-daemonset-8njtr"
Jul 24 21:51:36.817280 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:36.817246    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-664687df58-pzmzn_fb930e16-8f61-4e32-ae81-37a49ab4f923/manager/0.log"
Jul 24 21:51:37.013383 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:37.013315    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pm56h_906bf867-5ad4-4a5c-85fc-0724afe0f329/s3-init/0.log"
Jul 24 21:51:37.036575 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:37.036548    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pm56h_906bf867-5ad4-4a5c-85fc-0724afe0f329/download-hf-model/0.log"
Jul 24 21:51:42.038480 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:42.038451    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6_38bf3087-9101-4b71-be34-dd0726961e2a/kube-storage-version-migrator-operator/1.log"
Jul 24 21:51:42.039569 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:42.039548    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-x9jl6_38bf3087-9101-4b71-be34-dd0726961e2a/kube-storage-version-migrator-operator/0.log"
Jul 24 21:51:42.294057 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:42.293947    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qcfz_e02f6b0b-0d40-4e45-b729-bd64abdb2d64/konnectivity-agent/0.log" limit=1024
Jul 24 21:51:42.955642 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:42.955611    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5xnz9_1aafe0ca-0b4a-400d-add6-3998833dd481/kube-multus/0.log"
Jul 24 21:51:43.420363 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:43.420323    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pr7gn_a1be7ac6-f32e-40b4-8aa9-abca55380ae4/kube-multus-additional-cni-plugins/0.log"
Jul 24 21:51:43.448550 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:43.448524    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pr7gn_a1be7ac6-f32e-40b4-8aa9-abca55380ae4/egress-router-binary-copy/0.log"
Jul 24 21:51:43.475613 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:43.475588    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pr7gn_a1be7ac6-f32e-40b4-8aa9-abca55380ae4/cni-plugins/0.log"
Jul 24 21:51:43.504373 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:43.504353    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pr7gn_a1be7ac6-f32e-40b4-8aa9-abca55380ae4/bond-cni-plugin/0.log"
Jul 24 21:51:43.540611 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:43.540586    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pr7gn_a1be7ac6-f32e-40b4-8aa9-abca55380ae4/routeoverride-cni/0.log"
Jul 24 21:51:43.564262 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:43.564235    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pr7gn_a1be7ac6-f32e-40b4-8aa9-abca55380ae4/whereabouts-cni-bincopy/0.log"
Jul 24 21:51:43.592392 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:43.592363    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pr7gn_a1be7ac6-f32e-40b4-8aa9-abca55380ae4/whereabouts-cni/0.log"
Jul 24 21:51:43.679881 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:43.679801    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ldqjs_de44a92d-8440-4b1f-a0ba-479919425485/network-metrics-daemon/0.log"
Jul 24 21:51:43.700321 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:43.700297    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ldqjs_de44a92d-8440-4b1f-a0ba-479919425485/kube-rbac-proxy/0.log"
Jul 24 21:51:44.628509 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:44.628476    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-controller/0.log"
Jul 24 21:51:44.653563 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:44.653541    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/1.log"
Jul 24 21:51:44.653688 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:44.653538    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovn-acl-logging/0.log"
Jul 24 21:51:44.676698 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:44.676677    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/kube-rbac-proxy-node/0.log"
Jul 24 21:51:44.702437 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:44.702413    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/kube-rbac-proxy-ovn-metrics/0.log"
Jul 24 21:51:44.725741 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:44.725684    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/northd/0.log"
Jul 24 21:51:44.763456 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:44.763434    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/nbdb/0.log"
Jul 24 21:51:44.791412 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:44.791380    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/sbdb/0.log"
Jul 24 21:51:44.845620 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:44.845589    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-88v79_cb4843e9-e208-4283-bfb9-a5b4799e269f/ovnkube-controller/0.log"
Jul 24 21:51:46.410506 ip-10-0-136-232 kubenswrapper[2560]: I0724 21:51:46.410478    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-cpmnw_0f1880b1-1b8e-4633-841a-ed19c927a53d/network-check-target-container/0.log"